Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:57:55.766932 28095 test_util.cc:268] Using random seed: -293305732
I20240502 01:57:55.774163 28095 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:57:55.774288 28095 ts_itest-base.cc:113] --------------
I20240502 01:57:55.774322 28095 ts_itest-base.cc:114] 8 tablet servers
I20240502 01:57:55.774343 28095 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:57:55.774358 28095 ts_itest-base.cc:116] --------------
2024-05-02T01:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:57:55Z Disabled control of system clock
I20240502 01:57:55.783777 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.111.254:42667
--webserver_interface=127.27.111.254
--webserver_port=0
--builtin_ntp_servers=127.27.111.212:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.111.254:42667
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 01:57:55.892861 28107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:55.893115 28107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:55.893157 28107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:55.896140 28107 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:57:55.896203 28107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:57:55.896224 28107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:55.896250 28107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:57:55.896272 28107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:57:55.899928 28107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.111.254:42667
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.111.254:42667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.111.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.28107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:55.900771 28107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:55.901854 28107 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:55.908797 28112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:55.909345 28113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:55.909621 28115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:55.910256 28107 server_base.cc:1034] running on GCE node
I20240502 01:57:55.910769 28107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:55.911696 28107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:55.912853 28107 hybrid_clock.cc:648] HybridClock initialized: now 1714615075912830 us; error 37 us; skew 500 ppm
I20240502 01:57:55.914948 28107 webserver.cc:460] Webserver started at http://127.27.111.254:34011/ using document root <none> and password file <none>
I20240502 01:57:55.915500 28107 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:55.915575 28107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:55.915797 28107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:55.917757 28107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5d4a11e8419f4bc4a7b36523bbe92e69"
format_stamp: "Formatted at 2024-05-02 01:57:55 on dist-test-slave-0pw1"
I20240502 01:57:55.919968 28107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5d4a11e8419f4bc4a7b36523bbe92e69"
format_stamp: "Formatted at 2024-05-02 01:57:55 on dist-test-slave-0pw1"
I20240502 01:57:55.923851 28107 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:57:55.926052 28121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:55.927029 28107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 01:57:55.927136 28107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5d4a11e8419f4bc4a7b36523bbe92e69"
format_stamp: "Formatted at 2024-05-02 01:57:55 on dist-test-slave-0pw1"
I20240502 01:57:55.927220 28107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:55.966914 28107 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:55.967571 28107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:55.967726 28107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:55.974367 28107 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.254:42667
I20240502 01:57:55.974385 28174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.254:42667 every 8 connection(s)
I20240502 01:57:55.975569 28107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:57:55.978101 28175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:57:55.979439 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 28107
I20240502 01:57:55.979581 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:57:55.984246 28175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69: Bootstrap starting.
I20240502 01:57:55.986963 28175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69: Neither blocks nor log segments found. Creating new log.
I20240502 01:57:55.987665 28175 log.cc:824] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69: Log is configured to *not* fsync() on all Append() calls
I20240502 01:57:55.989563 28175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69: No bootstrap required, opened a new log
I20240502 01:57:55.992882 28175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4a11e8419f4bc4a7b36523bbe92e69" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 42667 } }
I20240502 01:57:55.993069 28175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:57:55.993115 28175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4a11e8419f4bc4a7b36523bbe92e69, State: Initialized, Role: FOLLOWER
I20240502 01:57:55.993546 28175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4a11e8419f4bc4a7b36523bbe92e69" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 42667 } }
I20240502 01:57:55.993675 28175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:57:55.993773 28175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:57:55.993906 28175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:57:55.994791 28175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4a11e8419f4bc4a7b36523bbe92e69" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 42667 } }
I20240502 01:57:55.995108 28175 leader_election.cc:302] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d4a11e8419f4bc4a7b36523bbe92e69; no voters:
I20240502 01:57:55.995381 28175 leader_election.cc:288] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:57:55.995760 28180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:57:55.995990 28180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [term 1 LEADER]: Becoming Leader. State: Replica: 5d4a11e8419f4bc4a7b36523bbe92e69, State: Running, Role: LEADER
I20240502 01:57:55.996307 28180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4a11e8419f4bc4a7b36523bbe92e69" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 42667 } }
I20240502 01:57:55.996577 28175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:57:55.997989 28181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d4a11e8419f4bc4a7b36523bbe92e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4a11e8419f4bc4a7b36523bbe92e69" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 42667 } } }
I20240502 01:57:55.998122 28181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [sys.catalog]: This master's current role is: LEADER
I20240502 01:57:55.998457 28182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d4a11e8419f4bc4a7b36523bbe92e69. Latest consensus state: current_term: 1 leader_uuid: "5d4a11e8419f4bc4a7b36523bbe92e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4a11e8419f4bc4a7b36523bbe92e69" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 42667 } } }
I20240502 01:57:55.998561 28182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69 [sys.catalog]: This master's current role is: LEADER
I20240502 01:57:55.998795 28189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:57:56.001238 28189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:57:56.005892 28189 catalog_manager.cc:1347] Generated new cluster ID: 2e20ff8e58d94e2da2d7cef0694f55ad
I20240502 01:57:56.005968 28189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:57:56.015908 28189 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:57:56.017171 28189 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:57:56.032176 28189 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5d4a11e8419f4bc4a7b36523bbe92e69: Generated new TSK 0
I20240502 01:57:56.033109 28189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:57:56.044970 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.193:0
--local_ip_for_outbound_sockets=127.27.111.193
--webserver_interface=127.27.111.193
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42667
--builtin_ntp_servers=127.27.111.212:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 01:57:56.155467 28199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:56.155709 28199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:56.155787 28199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:56.160409 28199 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:57:56.160518 28199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:56.160653 28199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.193
I20240502 01:57:56.165973 28199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.111.193
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.111.254:42667
--never_fsync=true
--heap_profile_path=/tmp/kudu.28199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.193
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:56.166977 28199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:56.168107 28199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:56.176479 28207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.177047 28199 server_base.cc:1034] running on GCE node
W20240502 01:57:56.177265 28204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.177284 28205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.177814 28199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:56.178454 28199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:56.179622 28199 hybrid_clock.cc:648] HybridClock initialized: now 1714615076179608 us; error 29 us; skew 500 ppm
I20240502 01:57:56.181598 28199 webserver.cc:460] Webserver started at http://127.27.111.193:35723/ using document root <none> and password file <none>
I20240502 01:57:56.182170 28199 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.182227 28199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.182385 28199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.184170 28199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a962820b582e4e158a2c765c544b8e52"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.184630 28199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a962820b582e4e158a2c765c544b8e52"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.188169 28199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240502 01:57:56.190438 28213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.191556 28199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:57:56.191676 28199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a962820b582e4e158a2c765c544b8e52"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.191763 28199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.204624 28199 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.205255 28199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:56.205399 28199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.205917 28199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:56.206863 28199 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:56.206918 28199 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.206957 28199 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:56.206980 28199 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.215242 28199 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.193:34973
I20240502 01:57:56.215303 28326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.193:34973 every 8 connection(s)
I20240502 01:57:56.216195 28199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:57:56.221261 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 28199
I20240502 01:57:56.221366 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:57:56.223572 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.194:0
--local_ip_for_outbound_sockets=127.27.111.194
--webserver_interface=127.27.111.194
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42667
--builtin_ntp_servers=127.27.111.212:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:57:56.225368 28327 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42667
I20240502 01:57:56.225594 28327 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.226100 28327 heartbeater.cc:511] Master 127.27.111.254:42667 requested a full tablet report, sending...
I20240502 01:57:56.227708 28139 ts_manager.cc:194] Registered new tserver with Master: a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:57:56.229529 28139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.193:47889
W20240502 01:57:56.333882 28330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:56.334142 28330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:56.334192 28330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:56.337266 28330 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:57:56.337349 28330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:56.337422 28330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.194
I20240502 01:57:56.341121 28330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.111.194
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.111.254:42667
--never_fsync=true
--heap_profile_path=/tmp/kudu.28330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.194
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:56.342059 28330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:56.343204 28330 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:56.351003 28335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.351157 28336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.351661 28338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.352458 28330 server_base.cc:1034] running on GCE node
I20240502 01:57:56.352880 28330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:56.353472 28330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:56.354625 28330 hybrid_clock.cc:648] HybridClock initialized: now 1714615076354613 us; error 31 us; skew 500 ppm
I20240502 01:57:56.356508 28330 webserver.cc:460] Webserver started at http://127.27.111.194:34057/ using document root <none> and password file <none>
I20240502 01:57:56.357075 28330 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.357137 28330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.357316 28330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.359042 28330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.359514 28330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.362845 28330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:57:56.365056 28344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.366055 28330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:56.366166 28330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.366251 28330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.383816 28330 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.384490 28330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:56.384666 28330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.385217 28330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:56.386133 28330 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:56.386185 28330 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.386243 28330 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:56.386272 28330 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.394656 28330 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.194:35369
I20240502 01:57:56.394708 28457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.194:35369 every 8 connection(s)
I20240502 01:57:56.395622 28330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:57:56.400995 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 28330
I20240502 01:57:56.401113 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:57:56.403509 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.195:0
--local_ip_for_outbound_sockets=127.27.111.195
--webserver_interface=127.27.111.195
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42667
--builtin_ntp_servers=127.27.111.212:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:57:56.404906 28458 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42667
I20240502 01:57:56.405185 28458 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.405858 28458 heartbeater.cc:511] Master 127.27.111.254:42667 requested a full tablet report, sending...
I20240502 01:57:56.406980 28139 ts_manager.cc:194] Registered new tserver with Master: 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369)
I20240502 01:57:56.407522 28139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.194:35457
W20240502 01:57:56.513095 28461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:56.513320 28461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:56.513381 28461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:56.516417 28461 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:57:56.516479 28461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:56.516570 28461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.195
I20240502 01:57:56.520040 28461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.111.195
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.111.254:42667
--never_fsync=true
--heap_profile_path=/tmp/kudu.28461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.195
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:56.520860 28461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:56.521939 28461 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:56.529817 28469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.529904 28467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.530118 28466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.530745 28461 server_base.cc:1034] running on GCE node
I20240502 01:57:56.531117 28461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:56.531639 28461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:56.532783 28461 hybrid_clock.cc:648] HybridClock initialized: now 1714615076532681 us; error 117 us; skew 500 ppm
I20240502 01:57:56.534750 28461 webserver.cc:460] Webserver started at http://127.27.111.195:39195/ using document root <none> and password file <none>
I20240502 01:57:56.535315 28461 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.535369 28461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.535560 28461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.537137 28461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ef3bc5c942d04bd29540a80415308353"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.537592 28461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ef3bc5c942d04bd29540a80415308353"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.540831 28461 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:57:56.542910 28475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.543871 28461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:56.543977 28461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ef3bc5c942d04bd29540a80415308353"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.544060 28461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.559116 28461 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.559757 28461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:56.559913 28461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.560434 28461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:56.561301 28461 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:56.561349 28461 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.561412 28461 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:56.561437 28461 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.569783 28461 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.195:46695
I20240502 01:57:56.569808 28588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.195:46695 every 8 connection(s)
I20240502 01:57:56.570752 28461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:57:56.579298 28589 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42667
I20240502 01:57:56.579515 28589 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.580005 28589 heartbeater.cc:511] Master 127.27.111.254:42667 requested a full tablet report, sending...
I20240502 01:57:56.580292 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 28461
I20240502 01:57:56.580390 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:57:56.580948 28139 ts_manager.cc:194] Registered new tserver with Master: ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695)
I20240502 01:57:56.581619 28139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.195:33167
I20240502 01:57:56.582540 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.196:0
--local_ip_for_outbound_sockets=127.27.111.196
--webserver_interface=127.27.111.196
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42667
--builtin_ntp_servers=127.27.111.212:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 01:57:56.688724 28592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:56.688985 28592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:56.689071 28592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:56.692188 28592 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:57:56.692274 28592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:56.692389 28592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.196
I20240502 01:57:56.696162 28592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.111.196
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.111.254:42667
--never_fsync=true
--heap_profile_path=/tmp/kudu.28592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.196
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:56.697206 28592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:56.698535 28592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:56.706720 28598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.706826 28600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.707213 28597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.707996 28592 server_base.cc:1034] running on GCE node
I20240502 01:57:56.708400 28592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:56.708971 28592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:56.710117 28592 hybrid_clock.cc:648] HybridClock initialized: now 1714615076710096 us; error 42 us; skew 500 ppm
I20240502 01:57:56.712050 28592 webserver.cc:460] Webserver started at http://127.27.111.196:43089/ using document root <none> and password file <none>
I20240502 01:57:56.712615 28592 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.712678 28592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.712870 28592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.714653 28592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "8cb8626af432419c922ceefff85f69f2"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.715160 28592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "8cb8626af432419c922ceefff85f69f2"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.718685 28592 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:57:56.720881 28606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.721901 28592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:56.722015 28592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8cb8626af432419c922ceefff85f69f2"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.722102 28592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.742736 28592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.743418 28592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:56.743584 28592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.744174 28592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:56.745087 28592 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:56.745136 28592 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.745198 28592 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:56.745224 28592 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.753765 28592 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.196:40797
I20240502 01:57:56.753803 28719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.196:40797 every 8 connection(s)
I20240502 01:57:56.754792 28592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:57:56.758056 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 28592
I20240502 01:57:56.758190 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:57:56.760625 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.197:0
--local_ip_for_outbound_sockets=127.27.111.197
--webserver_interface=127.27.111.197
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42667
--builtin_ntp_servers=127.27.111.212:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:57:56.764693 28720 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42667
I20240502 01:57:56.765008 28720 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.765792 28720 heartbeater.cc:511] Master 127.27.111.254:42667 requested a full tablet report, sending...
I20240502 01:57:56.767024 28139 ts_manager.cc:194] Registered new tserver with Master: 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:57:56.767861 28139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.196:53435
W20240502 01:57:56.869781 28723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:56.870028 28723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:56.870079 28723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:56.873376 28723 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:57:56.873452 28723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:56.873533 28723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.197
I20240502 01:57:56.877298 28723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.111.197
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.111.254:42667
--never_fsync=true
--heap_profile_path=/tmp/kudu.28723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.197
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:56.878183 28723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:56.879284 28723 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:56.887189 28731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.887521 28729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:56.887811 28728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:56.888506 28723 server_base.cc:1034] running on GCE node
I20240502 01:57:56.888903 28723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:56.889433 28723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:56.890571 28723 hybrid_clock.cc:648] HybridClock initialized: now 1714615076890555 us; error 34 us; skew 500 ppm
I20240502 01:57:56.892391 28723 webserver.cc:460] Webserver started at http://127.27.111.197:35433/ using document root <none> and password file <none>
I20240502 01:57:56.892930 28723 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:56.892985 28723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:56.893187 28723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:56.894912 28723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a1a0ca3ab1ca4872b2bb4395068b7033"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.895385 28723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a1a0ca3ab1ca4872b2bb4395068b7033"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.898903 28723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:57:56.901048 28737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.902014 28723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:56.902110 28723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a1a0ca3ab1ca4872b2bb4395068b7033"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:57:56.902200 28723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:56.926326 28723 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:56.927062 28723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:56.927227 28723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:56.927848 28723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:56.928771 28723 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:56.928818 28723 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.928856 28723 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:56.928871 28723 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:56.937299 28723 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.197:40085
I20240502 01:57:56.937326 28850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.197:40085 every 8 connection(s)
I20240502 01:57:56.938252 28723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:57:56.946655 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 28723
I20240502 01:57:56.946782 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:57:56.947008 28851 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42667
I20240502 01:57:56.947234 28851 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:56.947901 28851 heartbeater.cc:511] Master 127.27.111.254:42667 requested a full tablet report, sending...
I20240502 01:57:56.948909 28139 ts_manager.cc:194] Registered new tserver with Master: a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197:40085)
I20240502 01:57:56.948922 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.198:0
--local_ip_for_outbound_sockets=127.27.111.198
--webserver_interface=127.27.111.198
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42667
--builtin_ntp_servers=127.27.111.212:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:57:56.949512 28139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.197:52361
W20240502 01:57:57.062799 28854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:57.063068 28854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:57.063118 28854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:57.066252 28854 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:57:57.066327 28854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:57.066443 28854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.198
I20240502 01:57:57.070300 28854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.111.198
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.111.254:42667
--never_fsync=true
--heap_profile_path=/tmp/kudu.28854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.198
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:57.071269 28854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:57.072430 28854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:57.080518 28859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:57.080708 28862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:57.081071 28860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:57.081753 28854 server_base.cc:1034] running on GCE node
I20240502 01:57:57.082104 28854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:57.082695 28854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:57.083853 28854 hybrid_clock.cc:648] HybridClock initialized: now 1714615077083823 us; error 42 us; skew 500 ppm
I20240502 01:57:57.085870 28854 webserver.cc:460] Webserver started at http://127.27.111.198:41319/ using document root <none> and password file <none>
I20240502 01:57:57.086597 28854 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:57.086689 28854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:57.086900 28854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:57.088601 28854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "34a8610adef8401085b62d8fb1fbc8fe"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-0pw1"
I20240502 01:57:57.089079 28854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "34a8610adef8401085b62d8fb1fbc8fe"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-0pw1"
I20240502 01:57:57.092687 28854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:57:57.095023 28868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:57.096071 28854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:57:57.096175 28854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "34a8610adef8401085b62d8fb1fbc8fe"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-0pw1"
I20240502 01:57:57.096261 28854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:57.112566 28854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:57.113234 28854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:57.113400 28854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:57.113976 28854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:57.114908 28854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:57.114959 28854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:57.115018 28854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:57.115048 28854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:57.123852 28854 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.198:43065
I20240502 01:57:57.123888 28982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.198:43065 every 8 connection(s)
I20240502 01:57:57.124822 28854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:57:57.125488 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 28854
I20240502 01:57:57.125608 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:57:57.128118 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.199:0
--local_ip_for_outbound_sockets=127.27.111.199
--webserver_interface=127.27.111.199
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42667
--builtin_ntp_servers=127.27.111.212:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:57:57.136333 28983 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42667
I20240502 01:57:57.136612 28983 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:57.137369 28983 heartbeater.cc:511] Master 127.27.111.254:42667 requested a full tablet report, sending...
I20240502 01:57:57.138590 28139 ts_manager.cc:194] Registered new tserver with Master: 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198:43065)
I20240502 01:57:57.139400 28139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.198:47173
I20240502 01:57:57.232494 28327 heartbeater.cc:503] Master 127.27.111.254:42667 was elected leader, sending a full tablet report...
W20240502 01:57:57.243332 28986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:57.243558 28986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:57.243606 28986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:57.246752 28986 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:57:57.246820 28986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:57.246910 28986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.199
I20240502 01:57:57.250782 28986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.111.199
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.111.254:42667
--never_fsync=true
--heap_profile_path=/tmp/kudu.28986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.199
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:57.251705 28986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:57.252837 28986 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:57.260921 28991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:57.261155 28994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:57.261410 28992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:57.262204 28986 server_base.cc:1034] running on GCE node
I20240502 01:57:57.262634 28986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:57.263199 28986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:57.264381 28986 hybrid_clock.cc:648] HybridClock initialized: now 1714615077264358 us; error 38 us; skew 500 ppm
I20240502 01:57:57.266448 28986 webserver.cc:460] Webserver started at http://127.27.111.199:44385/ using document root <none> and password file <none>
I20240502 01:57:57.267024 28986 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:57.267088 28986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:57.267263 28986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:57.268949 28986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "dd5f234ed11745168cd67ee403ff098e"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-0pw1"
I20240502 01:57:57.269491 28986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "dd5f234ed11745168cd67ee403ff098e"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-0pw1"
I20240502 01:57:57.272953 28986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240502 01:57:57.275171 29000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:57.276130 28986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:57:57.276238 28986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "dd5f234ed11745168cd67ee403ff098e"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-0pw1"
I20240502 01:57:57.276320 28986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:57.302721 28986 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:57.303376 28986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:57.303525 28986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:57.304049 28986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:57.305004 28986 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:57.305056 28986 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:57.305096 28986 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:57.305119 28986 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:57.313505 28986 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.199:42203
I20240502 01:57:57.313525 29113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.199:42203 every 8 connection(s)
I20240502 01:57:57.314543 28986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:57:57.314803 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 28986
I20240502 01:57:57.314914 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:57:57.317371 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.200:0
--local_ip_for_outbound_sockets=127.27.111.200
--webserver_interface=127.27.111.200
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42667
--builtin_ntp_servers=127.27.111.212:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:57:57.324976 29114 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42667
I20240502 01:57:57.325268 29114 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:57.326069 29114 heartbeater.cc:511] Master 127.27.111.254:42667 requested a full tablet report, sending...
I20240502 01:57:57.327334 28139 ts_manager.cc:194] Registered new tserver with Master: dd5f234ed11745168cd67ee403ff098e (127.27.111.199:42203)
I20240502 01:57:57.327903 28139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.199:43581
I20240502 01:57:57.410012 28458 heartbeater.cc:503] Master 127.27.111.254:42667 was elected leader, sending a full tablet report...
W20240502 01:57:57.432977 29117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:57:57.433249 29117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:57:57.433306 29117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:57:57.436539 29117 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:57:57.436625 29117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:57:57.436733 29117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.200
I20240502 01:57:57.440627 29117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.111.200
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.111.254:42667
--never_fsync=true
--heap_profile_path=/tmp/kudu.29117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.200
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:57:57.441609 29117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:57:57.442795 29117 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:57:57.450712 29123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:57.451040 29122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:57:57.451320 29125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:57:57.452014 29117 server_base.cc:1034] running on GCE node
I20240502 01:57:57.452389 29117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:57:57.452944 29117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:57:57.454093 29117 hybrid_clock.cc:648] HybridClock initialized: now 1714615077454074 us; error 41 us; skew 500 ppm
I20240502 01:57:57.456049 29117 webserver.cc:460] Webserver started at http://127.27.111.200:39877/ using document root <none> and password file <none>
I20240502 01:57:57.456606 29117 fs_manager.cc:359] Metadata directory not provided
I20240502 01:57:57.456668 29117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:57:57.456840 29117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:57:57.458664 29117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "0773aa6e61f442bb86d966ae9e97cbc6"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-0pw1"
I20240502 01:57:57.459129 29117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "0773aa6e61f442bb86d966ae9e97cbc6"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-0pw1"
I20240502 01:57:57.462462 29117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:57:57.464653 29131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:57.465651 29117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 01:57:57.465759 29117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0773aa6e61f442bb86d966ae9e97cbc6"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-0pw1"
I20240502 01:57:57.465847 29117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:57:57.480332 29117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:57:57.481022 29117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:57:57.481190 29117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:57:57.481745 29117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:57:57.482697 29117 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:57:57.482749 29117 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:57.482806 29117 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:57:57.482831 29117 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:57:57.491449 29117 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.200:39123
I20240502 01:57:57.491464 29244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.200:39123 every 8 connection(s)
I20240502 01:57:57.492460 29117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:57:57.494849 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 29117
I20240502 01:57:57.494959 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:57:57.501842 29245 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42667
I20240502 01:57:57.502099 29245 heartbeater.cc:464] Registering TS with master...
I20240502 01:57:57.502672 29245 heartbeater.cc:511] Master 127.27.111.254:42667 requested a full tablet report, sending...
I20240502 01:57:57.503895 28139 ts_manager.cc:194] Registered new tserver with Master: 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200:39123)
I20240502 01:57:57.504464 28139 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.200:60685
I20240502 01:57:57.509461 28095 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:57:57.521195 28095 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:57:57.521291 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 28723
I20240502 01:57:57.527441 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 28854
I20240502 01:57:57.535409 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 28986
I20240502 01:57:57.540933 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 29117
I20240502 01:57:57.583516 28589 heartbeater.cc:503] Master 127.27.111.254:42667 was elected leader, sending a full tablet report...
I20240502 01:57:57.770550 28720 heartbeater.cc:503] Master 127.27.111.254:42667 was elected leader, sending a full tablet report...
I20240502 01:58:01.299043 28139 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52898:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:01.326040 28654 tablet_service.cc:1467] Processing CreateTablet for tablet 803991b0b27f470dbfe2397baa997621 (DEFAULT_TABLE table=rebalance_test_table_0 [id=efbe98a2708841018fae7c1b28c7a52d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.327113 28654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803991b0b27f470dbfe2397baa997621. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.327455 28653 tablet_service.cc:1467] Processing CreateTablet for tablet ba657ea33b9f445e88d3f1ea97c843a2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=efbe98a2708841018fae7c1b28c7a52d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.328250 28653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba657ea33b9f445e88d3f1ea97c843a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.328161 28391 tablet_service.cc:1467] Processing CreateTablet for tablet ba657ea33b9f445e88d3f1ea97c843a2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=efbe98a2708841018fae7c1b28c7a52d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.328495 28261 tablet_service.cc:1467] Processing CreateTablet for tablet 803991b0b27f470dbfe2397baa997621 (DEFAULT_TABLE table=rebalance_test_table_0 [id=efbe98a2708841018fae7c1b28c7a52d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.329036 28391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba657ea33b9f445e88d3f1ea97c843a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.329403 28261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803991b0b27f470dbfe2397baa997621. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.331130 28392 tablet_service.cc:1467] Processing CreateTablet for tablet a14a4e6aa65240a4a1ea29bd564521ab (DEFAULT_TABLE table=rebalance_test_table_0 [id=efbe98a2708841018fae7c1b28c7a52d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.331370 28392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14a4e6aa65240a4a1ea29bd564521ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.334559 28260 tablet_service.cc:1467] Processing CreateTablet for tablet a14a4e6aa65240a4a1ea29bd564521ab (DEFAULT_TABLE table=rebalance_test_table_0 [id=efbe98a2708841018fae7c1b28c7a52d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.334875 28260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14a4e6aa65240a4a1ea29bd564521ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.338299 29265 tablet_bootstrap.cc:492] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2: Bootstrap starting.
I20240502 01:58:01.339068 28259 tablet_service.cc:1467] Processing CreateTablet for tablet ba657ea33b9f445e88d3f1ea97c843a2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=efbe98a2708841018fae7c1b28c7a52d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.339792 28259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba657ea33b9f445e88d3f1ea97c843a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.340847 29265 tablet_bootstrap.cc:654] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.341523 29265 log.cc:824] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:01.342725 29266 tablet_bootstrap.cc:492] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4: Bootstrap starting.
I20240502 01:58:01.345046 29266 tablet_bootstrap.cc:654] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.345499 29265 tablet_bootstrap.cc:492] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2: No bootstrap required, opened a new log
I20240502 01:58:01.345917 29267 tablet_bootstrap.cc:492] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:01.346005 29266 log.cc:824] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:01.348258 29266 tablet_bootstrap.cc:492] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4: No bootstrap required, opened a new log
I20240502 01:58:01.348435 29266 ts_tablet_manager.cc:1367] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240502 01:58:01.350070 29265 ts_tablet_manager.cc:1367] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240502 01:58:01.349850 28522 tablet_service.cc:1467] Processing CreateTablet for tablet a14a4e6aa65240a4a1ea29bd564521ab (DEFAULT_TABLE table=rebalance_test_table_0 [id=efbe98a2708841018fae7c1b28c7a52d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.350955 28522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14a4e6aa65240a4a1ea29bd564521ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.352237 29266 raft_consensus.cc:348] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:01.352442 29266 raft_consensus.cc:374] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.352500 29266 raft_consensus.cc:729] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.352988 29266 consensus_queue.cc:260] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:01.353940 29266 ts_tablet_manager.cc:1398] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240502 01:58:01.354319 29266 tablet_bootstrap.cc:492] T ba657ea33b9f445e88d3f1ea97c843a2 P 8f4edf52bb2d4624ae596b032e50c2a4: Bootstrap starting.
I20240502 01:58:01.349857 28523 tablet_service.cc:1467] Processing CreateTablet for tablet 803991b0b27f470dbfe2397baa997621 (DEFAULT_TABLE table=rebalance_test_table_0 [id=efbe98a2708841018fae7c1b28c7a52d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.355152 28523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803991b0b27f470dbfe2397baa997621. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.355351 29266 tablet_bootstrap.cc:654] T ba657ea33b9f445e88d3f1ea97c843a2 P 8f4edf52bb2d4624ae596b032e50c2a4: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.358235 29266 tablet_bootstrap.cc:492] T ba657ea33b9f445e88d3f1ea97c843a2 P 8f4edf52bb2d4624ae596b032e50c2a4: No bootstrap required, opened a new log
I20240502 01:58:01.358321 29266 ts_tablet_manager.cc:1367] T ba657ea33b9f445e88d3f1ea97c843a2 P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:01.358887 29266 raft_consensus.cc:348] T ba657ea33b9f445e88d3f1ea97c843a2 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.359025 29266 raft_consensus.cc:374] T ba657ea33b9f445e88d3f1ea97c843a2 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.359057 29266 raft_consensus.cc:729] T ba657ea33b9f445e88d3f1ea97c843a2 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.359180 29266 consensus_queue.cc:260] T ba657ea33b9f445e88d3f1ea97c843a2 P 8f4edf52bb2d4624ae596b032e50c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.359432 29266 ts_tablet_manager.cc:1398] T ba657ea33b9f445e88d3f1ea97c843a2 P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:01.359967 29272 tablet_bootstrap.cc:492] T a14a4e6aa65240a4a1ea29bd564521ab P ef3bc5c942d04bd29540a80415308353: Bootstrap starting.
I20240502 01:58:01.361709 29267 tablet_bootstrap.cc:654] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.362129 29272 tablet_bootstrap.cc:654] T a14a4e6aa65240a4a1ea29bd564521ab P ef3bc5c942d04bd29540a80415308353: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.362795 29267 log.cc:824] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:01.362964 29272 log.cc:824] T a14a4e6aa65240a4a1ea29bd564521ab P ef3bc5c942d04bd29540a80415308353: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:01.363305 29265 raft_consensus.cc:348] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.363600 29265 raft_consensus.cc:374] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.363694 29265 raft_consensus.cc:729] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.364195 29265 consensus_queue.cc:260] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.364760 29272 tablet_bootstrap.cc:492] T a14a4e6aa65240a4a1ea29bd564521ab P ef3bc5c942d04bd29540a80415308353: No bootstrap required, opened a new log
I20240502 01:58:01.364856 29267 tablet_bootstrap.cc:492] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52: No bootstrap required, opened a new log
I20240502 01:58:01.364949 29272 ts_tablet_manager.cc:1367] T a14a4e6aa65240a4a1ea29bd564521ab P ef3bc5c942d04bd29540a80415308353: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:58:01.365044 29267 ts_tablet_manager.cc:1367] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20240502 01:58:01.365170 29265 ts_tablet_manager.cc:1398] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20240502 01:58:01.368192 29265 tablet_bootstrap.cc:492] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2: Bootstrap starting.
I20240502 01:58:01.368918 29272 raft_consensus.cc:348] T a14a4e6aa65240a4a1ea29bd564521ab P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:01.369169 29272 raft_consensus.cc:374] T a14a4e6aa65240a4a1ea29bd564521ab P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.369261 29272 raft_consensus.cc:729] T a14a4e6aa65240a4a1ea29bd564521ab P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.369545 29265 tablet_bootstrap.cc:654] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.369866 29272 consensus_queue.cc:260] T a14a4e6aa65240a4a1ea29bd564521ab P ef3bc5c942d04bd29540a80415308353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:01.370959 29272 ts_tablet_manager.cc:1398] T a14a4e6aa65240a4a1ea29bd564521ab P ef3bc5c942d04bd29540a80415308353: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20240502 01:58:01.370849 29267 raft_consensus.cc:348] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:01.371136 29267 raft_consensus.cc:374] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.371218 29267 raft_consensus.cc:729] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.371852 29267 consensus_queue.cc:260] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:01.372400 29272 tablet_bootstrap.cc:492] T 803991b0b27f470dbfe2397baa997621 P ef3bc5c942d04bd29540a80415308353: Bootstrap starting.
I20240502 01:58:01.373445 29272 tablet_bootstrap.cc:654] T 803991b0b27f470dbfe2397baa997621 P ef3bc5c942d04bd29540a80415308353: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.374814 29265 tablet_bootstrap.cc:492] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2: No bootstrap required, opened a new log
I20240502 01:58:01.374914 29265 ts_tablet_manager.cc:1367] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240502 01:58:01.375222 29272 tablet_bootstrap.cc:492] T 803991b0b27f470dbfe2397baa997621 P ef3bc5c942d04bd29540a80415308353: No bootstrap required, opened a new log
I20240502 01:58:01.375293 29272 ts_tablet_manager.cc:1367] T 803991b0b27f470dbfe2397baa997621 P ef3bc5c942d04bd29540a80415308353: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:01.375530 29265 raft_consensus.cc:348] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.375643 29265 raft_consensus.cc:374] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.375675 29265 raft_consensus.cc:729] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.375795 29265 consensus_queue.cc:260] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.375772 29272 raft_consensus.cc:348] T 803991b0b27f470dbfe2397baa997621 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.375883 29272 raft_consensus.cc:374] T 803991b0b27f470dbfe2397baa997621 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.375924 29272 raft_consensus.cc:729] T 803991b0b27f470dbfe2397baa997621 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.376078 29265 ts_tablet_manager.cc:1398] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:01.376039 29272 consensus_queue.cc:260] T 803991b0b27f470dbfe2397baa997621 P ef3bc5c942d04bd29540a80415308353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.376296 29272 ts_tablet_manager.cc:1398] T 803991b0b27f470dbfe2397baa997621 P ef3bc5c942d04bd29540a80415308353: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:01.377285 29267 ts_tablet_manager.cc:1398] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240502 01:58:01.377605 29267 tablet_bootstrap.cc:492] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:01.380836 29267 tablet_bootstrap.cc:654] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.381306 29270 raft_consensus.cc:482] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:01.381489 29270 raft_consensus.cc:504] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:01.382752 29267 tablet_bootstrap.cc:492] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52: No bootstrap required, opened a new log
I20240502 01:58:01.382856 29267 ts_tablet_manager.cc:1367] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:01.382848 29270 leader_election.cc:288] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695), a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:58:01.383700 29267 raft_consensus.cc:348] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.383841 29267 raft_consensus.cc:374] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.383880 29267 raft_consensus.cc:729] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.384040 29267 consensus_queue.cc:260] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.384837 29267 ts_tablet_manager.cc:1398] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:01.385072 29267 tablet_bootstrap.cc:492] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:01.386286 29267 tablet_bootstrap.cc:654] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.387806 29267 tablet_bootstrap.cc:492] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52: No bootstrap required, opened a new log
I20240502 01:58:01.387912 29267 ts_tablet_manager.cc:1367] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:58:01.388513 29267 raft_consensus.cc:348] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.388651 29267 raft_consensus.cc:374] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.388701 29267 raft_consensus.cc:729] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.388861 29267 consensus_queue.cc:260] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.389503 29267 ts_tablet_manager.cc:1398] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:58:01.393468 28543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14a4e6aa65240a4a1ea29bd564521ab" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353" is_pre_election: true
I20240502 01:58:01.393908 28543 raft_consensus.cc:2454] T a14a4e6aa65240a4a1ea29bd564521ab P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 0.
I20240502 01:58:01.394381 28347 leader_election.cc:302] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f4edf52bb2d4624ae596b032e50c2a4, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:01.394692 29270 raft_consensus.cc:2781] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:01.394770 29270 raft_consensus.cc:482] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:01.394814 29270 raft_consensus.cc:3037] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:01.395985 29270 raft_consensus.cc:504] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:01.396339 29270 leader_election.cc:288] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 election: Requested vote from peers ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695), a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
W20240502 01:58:01.400020 28459 tablet.cc:2369] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:01.402576 28543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14a4e6aa65240a4a1ea29bd564521ab" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353"
I20240502 01:58:01.402757 28543 raft_consensus.cc:3037] T a14a4e6aa65240a4a1ea29bd564521ab P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:01.403323 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14a4e6aa65240a4a1ea29bd564521ab" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52"
I20240502 01:58:01.403578 28281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14a4e6aa65240a4a1ea29bd564521ab" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52" is_pre_election: true
I20240502 01:58:01.403594 28280 raft_consensus.cc:3037] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:01.404229 28543 raft_consensus.cc:2454] T a14a4e6aa65240a4a1ea29bd564521ab P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 1.
I20240502 01:58:01.404742 28280 raft_consensus.cc:2454] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 1.
I20240502 01:58:01.405012 28347 leader_election.cc:302] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f4edf52bb2d4624ae596b032e50c2a4, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:01.405241 29270 raft_consensus.cc:2781] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:01.405475 29270 raft_consensus.cc:686] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Running, Role: LEADER
I20240502 01:58:01.406061 29270 consensus_queue.cc:237] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:01.409642 28137 catalog_manager.cc:5484] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:01.423436 29275 raft_consensus.cc:482] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:01.423667 29275 raft_consensus.cc:504] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.424799 29275 leader_election.cc:288] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369), a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:58:01.429294 28412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba657ea33b9f445e88d3f1ea97c843a2" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" is_pre_election: true
I20240502 01:58:01.429535 28412 raft_consensus.cc:2454] T ba657ea33b9f445e88d3f1ea97c843a2 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 0.
I20240502 01:58:01.429983 28607 leader_election.cc:302] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, 8f4edf52bb2d4624ae596b032e50c2a4; no voters:
I20240502 01:58:01.430320 29275 raft_consensus.cc:2781] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:01.430405 29275 raft_consensus.cc:482] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:01.430476 29275 raft_consensus.cc:3037] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:01.431180 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba657ea33b9f445e88d3f1ea97c843a2" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52" is_pre_election: true
I20240502 01:58:01.431357 28280 raft_consensus.cc:2454] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 0.
I20240502 01:58:01.431645 29275 raft_consensus.cc:504] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.432016 29275 leader_election.cc:288] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 election: Requested vote from peers 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369), a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:58:01.432524 28412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba657ea33b9f445e88d3f1ea97c843a2" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
I20240502 01:58:01.432567 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba657ea33b9f445e88d3f1ea97c843a2" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52"
I20240502 01:58:01.432647 28412 raft_consensus.cc:3037] T ba657ea33b9f445e88d3f1ea97c843a2 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:01.432709 28280 raft_consensus.cc:3037] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:01.433439 28412 raft_consensus.cc:2454] T ba657ea33b9f445e88d3f1ea97c843a2 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 1.
I20240502 01:58:01.433892 28280 raft_consensus.cc:2454] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 1.
I20240502 01:58:01.434258 28607 leader_election.cc:302] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, a962820b582e4e158a2c765c544b8e52; no voters:
I20240502 01:58:01.434510 29275 raft_consensus.cc:2781] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:01.434746 29275 raft_consensus.cc:686] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [term 1 LEADER]: Becoming Leader. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Running, Role: LEADER
I20240502 01:58:01.435089 29275 consensus_queue.cc:237] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.438764 28137 catalog_manager.cc:5484] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cb8626af432419c922ceefff85f69f2 (127.27.111.196). New cstate: current_term: 1 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:01.471251 28328 tablet.cc:2369] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:58:01.509493 28721 tablet.cc:2369] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:58:01.575163 28590 tablet.cc:2369] T 803991b0b27f470dbfe2397baa997621 P ef3bc5c942d04bd29540a80415308353: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:01.623169 29275 raft_consensus.cc:482] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:01.623317 29275 raft_consensus.cc:504] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.623814 29275 leader_election.cc:288] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973), ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695)
I20240502 01:58:01.624397 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803991b0b27f470dbfe2397baa997621" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52" is_pre_election: true
I20240502 01:58:01.624567 28280 raft_consensus.cc:2454] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 0.
I20240502 01:58:01.624884 28607 leader_election.cc:302] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, a962820b582e4e158a2c765c544b8e52; no voters:
I20240502 01:58:01.625054 29275 raft_consensus.cc:2781] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:01.625110 29275 raft_consensus.cc:482] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:01.625159 29275 raft_consensus.cc:3037] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:01.626235 29275 raft_consensus.cc:504] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.626608 29275 leader_election.cc:288] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 election: Requested vote from peers a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973), ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695)
I20240502 01:58:01.626957 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803991b0b27f470dbfe2397baa997621" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52"
I20240502 01:58:01.627084 28280 raft_consensus.cc:3037] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:01.627732 28543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803991b0b27f470dbfe2397baa997621" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353" is_pre_election: true
I20240502 01:58:01.627782 28542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803991b0b27f470dbfe2397baa997621" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353"
I20240502 01:58:01.627874 28543 raft_consensus.cc:2454] T 803991b0b27f470dbfe2397baa997621 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 0.
I20240502 01:58:01.628050 28280 raft_consensus.cc:2454] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 1.
I20240502 01:58:01.628351 28607 leader_election.cc:302] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, a962820b582e4e158a2c765c544b8e52; no voters:
I20240502 01:58:01.628530 29275 raft_consensus.cc:2781] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:01.628618 29275 raft_consensus.cc:686] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [term 1 LEADER]: Becoming Leader. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Running, Role: LEADER
I20240502 01:58:01.628754 29275 consensus_queue.cc:237] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.630360 28137 catalog_manager.cc:5484] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cb8626af432419c922ceefff85f69f2 (127.27.111.196). New cstate: current_term: 1 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:01.804296 29291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:01.804548 29291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:01.807602 29291 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:58:01.819793 29270 consensus_queue.cc:1035] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:01.825937 29304 consensus_queue.cc:1035] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 01:58:01.896462 28542 raft_consensus.cc:3037] T 803991b0b27f470dbfe2397baa997621 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:01.896852 28280 raft_consensus.cc:1261] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:01.897209 28281 raft_consensus.cc:1261] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:01.897550 29288 consensus_queue.cc:1035] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:01.897655 28412 raft_consensus.cc:1261] T ba657ea33b9f445e88d3f1ea97c843a2 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:01.897719 28542 raft_consensus.cc:1261] T 803991b0b27f470dbfe2397baa997621 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:01.898157 29288 consensus_queue.cc:1035] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:01.907835 29275 consensus_queue.cc:1035] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:01.908354 29275 consensus_queue.cc:1035] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:01.910465 29328 mvcc.cc:205] Tried to move back new op lower bound from 7023063375441702912 to 7023063374352568320. Current Snapshot: MvccSnapshot[applied={T|T < 7023063375441702912}]
I20240502 01:58:01.919519 29305 mvcc.cc:205] Tried to move back new op lower bound from 7023063375441702912 to 7023063374352568320. Current Snapshot: MvccSnapshot[applied={T|T < 7023063375441702912}]
INSERT report
rows total: 2000
time total: 100.396 ms
time per row: 0.0501982 ms
I20240502 01:58:01.954823 29340 mvcc.cc:205] Tried to move back new op lower bound from 7023063375471325184 to 7023063373561196544. Current Snapshot: MvccSnapshot[applied={T|T < 7023063375446761472}]
I20240502 01:58:01.960808 28131 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52898:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:01.966130 28392 tablet_service.cc:1467] Processing CreateTablet for tablet 43d194188c6f455aab0099df8ed0cbbc (DEFAULT_TABLE table=rebalance_test_table_1 [id=eae4e7ef163a4d2788386ab9c9fd7841]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.966495 28392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43d194188c6f455aab0099df8ed0cbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.966372 28523 tablet_service.cc:1467] Processing CreateTablet for tablet 43d194188c6f455aab0099df8ed0cbbc (DEFAULT_TABLE table=rebalance_test_table_1 [id=eae4e7ef163a4d2788386ab9c9fd7841]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.966918 28523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43d194188c6f455aab0099df8ed0cbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.967126 28653 tablet_service.cc:1467] Processing CreateTablet for tablet 43d194188c6f455aab0099df8ed0cbbc (DEFAULT_TABLE table=rebalance_test_table_1 [id=eae4e7ef163a4d2788386ab9c9fd7841]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.967375 28653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43d194188c6f455aab0099df8ed0cbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.967671 28261 tablet_service.cc:1467] Processing CreateTablet for tablet 2efc1c7671084686ab57f6943aa60134 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eae4e7ef163a4d2788386ab9c9fd7841]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.967939 28261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2efc1c7671084686ab57f6943aa60134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.966611 28522 tablet_service.cc:1467] Processing CreateTablet for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c (DEFAULT_TABLE table=rebalance_test_table_1 [id=eae4e7ef163a4d2788386ab9c9fd7841]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.969321 28522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.969301 28392 tablet_service.cc:1467] Processing CreateTablet for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c (DEFAULT_TABLE table=rebalance_test_table_1 [id=eae4e7ef163a4d2788386ab9c9fd7841]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.969579 28392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.969772 28391 tablet_service.cc:1467] Processing CreateTablet for tablet 2efc1c7671084686ab57f6943aa60134 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eae4e7ef163a4d2788386ab9c9fd7841]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.969996 28391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2efc1c7671084686ab57f6943aa60134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.967319 28654 tablet_service.cc:1467] Processing CreateTablet for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c (DEFAULT_TABLE table=rebalance_test_table_1 [id=eae4e7ef163a4d2788386ab9c9fd7841]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.970510 28654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.971697 29349 tablet_bootstrap.cc:492] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4: Bootstrap starting.
I20240502 01:58:01.972143 28522 tablet_service.cc:1467] Processing CreateTablet for tablet 2efc1c7671084686ab57f6943aa60134 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eae4e7ef163a4d2788386ab9c9fd7841]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:01.972362 28522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2efc1c7671084686ab57f6943aa60134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:01.972589 29351 tablet_bootstrap.cc:492] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2: Bootstrap starting.
I20240502 01:58:01.972762 29349 tablet_bootstrap.cc:654] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.973522 29350 tablet_bootstrap.cc:492] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353: Bootstrap starting.
I20240502 01:58:01.973670 29351 tablet_bootstrap.cc:654] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.974440 29350 tablet_bootstrap.cc:654] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.975037 29351 tablet_bootstrap.cc:492] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2: No bootstrap required, opened a new log
I20240502 01:58:01.975075 29352 tablet_bootstrap.cc:492] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:01.975186 29351 ts_tablet_manager.cc:1367] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:01.975364 29349 tablet_bootstrap.cc:492] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4: No bootstrap required, opened a new log
I20240502 01:58:01.975502 29349 ts_tablet_manager.cc:1367] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:01.975729 29351 raft_consensus.cc:348] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.975908 29351 raft_consensus.cc:374] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.975999 29352 tablet_bootstrap.cc:654] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.975991 29351 raft_consensus.cc:729] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.976025 29349 raft_consensus.cc:348] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.976197 29349 raft_consensus.cc:374] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.976171 29351 consensus_queue.cc:260] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.976298 29349 raft_consensus.cc:729] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.976512 29351 ts_tablet_manager.cc:1398] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:01.976670 29351 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2: Bootstrap starting.
I20240502 01:58:01.976877 29349 consensus_queue.cc:260] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.977367 29349 ts_tablet_manager.cc:1398] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:01.977581 29349 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4: Bootstrap starting.
I20240502 01:58:01.977878 29352 tablet_bootstrap.cc:492] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52: No bootstrap required, opened a new log
I20240502 01:58:01.977963 29352 ts_tablet_manager.cc:1367] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:01.978348 29352 raft_consensus.cc:348] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.978490 29352 raft_consensus.cc:374] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.978525 29352 raft_consensus.cc:729] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.978633 29352 consensus_queue.cc:260] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.978863 29352 ts_tablet_manager.cc:1398] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:01.979920 29351 tablet_bootstrap.cc:654] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.980347 29349 tablet_bootstrap.cc:654] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.980924 29351 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2: No bootstrap required, opened a new log
I20240502 01:58:01.981003 29351 ts_tablet_manager.cc:1367] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:01.981370 29351 raft_consensus.cc:348] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.981462 29351 raft_consensus.cc:374] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.981487 29351 raft_consensus.cc:729] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.981582 29351 consensus_queue.cc:260] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.981791 29350 tablet_bootstrap.cc:492] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353: No bootstrap required, opened a new log
I20240502 01:58:01.981789 29351 ts_tablet_manager.cc:1398] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:01.981892 29350 ts_tablet_manager.cc:1367] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240502 01:58:01.982352 29350 raft_consensus.cc:348] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.982651 29350 raft_consensus.cc:374] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.982694 29350 raft_consensus.cc:729] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.982817 29350 consensus_queue.cc:260] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.983029 29349 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4: No bootstrap required, opened a new log
I20240502 01:58:01.983098 29349 ts_tablet_manager.cc:1367] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:01.983115 29350 ts_tablet_manager.cc:1398] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:01.983294 29350 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353: Bootstrap starting.
I20240502 01:58:01.983573 29349 raft_consensus.cc:348] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.983661 29349 raft_consensus.cc:374] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.983687 29349 raft_consensus.cc:729] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.983794 29349 consensus_queue.cc:260] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.984042 29349 ts_tablet_manager.cc:1398] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:01.984133 29350 tablet_bootstrap.cc:654] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.984174 29349 tablet_bootstrap.cc:492] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4: Bootstrap starting.
I20240502 01:58:01.984964 29349 tablet_bootstrap.cc:654] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.986166 29349 tablet_bootstrap.cc:492] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4: No bootstrap required, opened a new log
I20240502 01:58:01.986258 29349 ts_tablet_manager.cc:1367] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:01.986621 29350 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353: No bootstrap required, opened a new log
I20240502 01:58:01.986699 29350 ts_tablet_manager.cc:1367] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:01.986662 29349 raft_consensus.cc:348] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.986760 29349 raft_consensus.cc:374] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.986790 29349 raft_consensus.cc:729] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.986901 29349 consensus_queue.cc:260] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.987130 29349 ts_tablet_manager.cc:1398] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:01.987488 29350 raft_consensus.cc:348] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.987627 29350 raft_consensus.cc:374] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.987665 29350 raft_consensus.cc:729] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.987854 29350 consensus_queue.cc:260] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:01.988200 29350 ts_tablet_manager.cc:1398] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:01.988401 29350 tablet_bootstrap.cc:492] T 2efc1c7671084686ab57f6943aa60134 P ef3bc5c942d04bd29540a80415308353: Bootstrap starting.
I20240502 01:58:01.989287 29350 tablet_bootstrap.cc:654] T 2efc1c7671084686ab57f6943aa60134 P ef3bc5c942d04bd29540a80415308353: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:01.990613 29350 tablet_bootstrap.cc:492] T 2efc1c7671084686ab57f6943aa60134 P ef3bc5c942d04bd29540a80415308353: No bootstrap required, opened a new log
I20240502 01:58:01.990682 29350 ts_tablet_manager.cc:1367] T 2efc1c7671084686ab57f6943aa60134 P ef3bc5c942d04bd29540a80415308353: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:58:01.991017 29350 raft_consensus.cc:348] T 2efc1c7671084686ab57f6943aa60134 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.991101 29350 raft_consensus.cc:374] T 2efc1c7671084686ab57f6943aa60134 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:01.991135 29350 raft_consensus.cc:729] T 2efc1c7671084686ab57f6943aa60134 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Initialized, Role: FOLLOWER
I20240502 01:58:01.991257 29350 consensus_queue.cc:260] T 2efc1c7671084686ab57f6943aa60134 P ef3bc5c942d04bd29540a80415308353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:01.991464 29350 ts_tablet_manager.cc:1398] T 2efc1c7671084686ab57f6943aa60134 P ef3bc5c942d04bd29540a80415308353: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:02.173977 29284 raft_consensus.cc:482] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:02.174080 29276 raft_consensus.cc:482] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:02.174134 29284 raft_consensus.cc:504] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:02.174196 29276 raft_consensus.cc:504] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.174661 29284 leader_election.cc:288] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973), ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695)
I20240502 01:58:02.175163 28542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2efc1c7671084686ab57f6943aa60134" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353" is_pre_election: true
I20240502 01:58:02.175303 28542 raft_consensus.cc:2454] T 2efc1c7671084686ab57f6943aa60134 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 0.
I20240502 01:58:02.175380 29276 leader_election.cc:288] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369), 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:02.175387 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2efc1c7671084686ab57f6943aa60134" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52" is_pre_election: true
I20240502 01:58:02.175562 28280 raft_consensus.cc:2454] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 0.
I20240502 01:58:02.175992 28345 leader_election.cc:302] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f4edf52bb2d4624ae596b032e50c2a4, a962820b582e4e158a2c765c544b8e52; no voters:
I20240502 01:58:02.176195 29284 raft_consensus.cc:2781] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:02.176257 29284 raft_consensus.cc:482] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:02.176285 29284 raft_consensus.cc:3037] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.177405 29284 raft_consensus.cc:504] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:02.177789 29284 leader_election.cc:288] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 election: Requested vote from peers a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973), ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695)
I20240502 01:58:02.178145 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2efc1c7671084686ab57f6943aa60134" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52"
I20240502 01:58:02.178265 28280 raft_consensus.cc:3037] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.178315 28542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2efc1c7671084686ab57f6943aa60134" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353"
I20240502 01:58:02.178478 28542 raft_consensus.cc:3037] T 2efc1c7671084686ab57f6943aa60134 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.179311 28280 raft_consensus.cc:2454] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 1.
I20240502 01:58:02.179487 28542 raft_consensus.cc:2454] T 2efc1c7671084686ab57f6943aa60134 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 1.
I20240502 01:58:02.179632 28345 leader_election.cc:302] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f4edf52bb2d4624ae596b032e50c2a4, a962820b582e4e158a2c765c544b8e52; no voters:
I20240502 01:58:02.179790 29284 raft_consensus.cc:2781] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:02.179865 29284 raft_consensus.cc:686] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Running, Role: LEADER
I20240502 01:58:02.180006 29284 consensus_queue.cc:237] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:02.180421 28674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d194188c6f455aab0099df8ed0cbbc" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2" is_pre_election: true
I20240502 01:58:02.180653 28674 raft_consensus.cc:2454] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 0.
I20240502 01:58:02.180967 28412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d194188c6f455aab0099df8ed0cbbc" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" is_pre_election: true
I20240502 01:58:02.181077 28412 raft_consensus.cc:2454] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 0.
I20240502 01:58:02.181108 28479 leader_election.cc:302] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:02.181361 29276 raft_consensus.cc:2781] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:02.181450 29276 raft_consensus.cc:482] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:02.181491 29276 raft_consensus.cc:3037] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.182073 28131 catalog_manager.cc:5484] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:02.182477 29276 raft_consensus.cc:504] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.182837 29276 leader_election.cc:288] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 election: Requested vote from peers 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369), 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:02.183177 28674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d194188c6f455aab0099df8ed0cbbc" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2"
I20240502 01:58:02.183297 28674 raft_consensus.cc:3037] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.183954 28412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d194188c6f455aab0099df8ed0cbbc" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
I20240502 01:58:02.184063 28412 raft_consensus.cc:3037] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.184324 28674 raft_consensus.cc:2454] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 1.
I20240502 01:58:02.184593 28479 leader_election.cc:302] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:02.184756 29276 raft_consensus.cc:2781] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:02.184989 29276 raft_consensus.cc:686] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Becoming Leader. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Running, Role: LEADER
I20240502 01:58:02.185036 28412 raft_consensus.cc:2454] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 1.
I20240502 01:58:02.185314 29276 consensus_queue.cc:237] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.187556 28131 catalog_manager.cc:5484] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef3bc5c942d04bd29540a80415308353 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "ef3bc5c942d04bd29540a80415308353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:02.230883 29304 raft_consensus.cc:482] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:02.231067 29304 raft_consensus.cc:504] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.231606 29304 leader_election.cc:288] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695), 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:02.232086 28542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353" is_pre_election: true
I20240502 01:58:02.232236 28542 raft_consensus.cc:2454] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 0.
I20240502 01:58:02.232581 28347 leader_election.cc:302] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f4edf52bb2d4624ae596b032e50c2a4, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:02.232774 29304 raft_consensus.cc:2781] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:02.232844 29304 raft_consensus.cc:482] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:02.232890 29304 raft_consensus.cc:3037] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.233790 29304 raft_consensus.cc:504] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.234163 29304 leader_election.cc:288] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 election: Requested vote from peers ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695), 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:02.234513 28542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353"
I20240502 01:58:02.234644 28542 raft_consensus.cc:3037] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.235530 28542 raft_consensus.cc:2454] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 1.
I20240502 01:58:02.235785 28674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2" is_pre_election: true
I20240502 01:58:02.235841 28347 leader_election.cc:302] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f4edf52bb2d4624ae596b032e50c2a4, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:02.235791 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2"
I20240502 01:58:02.235946 28674 raft_consensus.cc:2454] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 0.
I20240502 01:58:02.236002 29304 raft_consensus.cc:2781] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:02.236070 29304 raft_consensus.cc:686] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Running, Role: LEADER
I20240502 01:58:02.236202 29304 consensus_queue.cc:237] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.237937 28131 catalog_manager.cc:5484] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:02.383129 29369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:02.383416 29369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:02.386554 29369 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 01:58:02.468365 28280 raft_consensus.cc:1261] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:02.468348 28542 raft_consensus.cc:1261] T 2efc1c7671084686ab57f6943aa60134 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:02.469202 29304 consensus_queue.cc:1035] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:02.469789 28673 raft_consensus.cc:3037] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.469751 29284 consensus_queue.cc:1035] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:02.471136 28673 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:02.472741 29284 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:02.473027 28543 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:02.473434 29284 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:02.476505 28412 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:02.477069 29276 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:02.478284 28674 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:02.478693 29365 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:02.487195 29400 mvcc.cc:205] Tried to move back new op lower bound from 7023063377784295424 to 7023063376610697216. Current Snapshot: MvccSnapshot[applied={T|T < 7023063377784295424}]
INSERT report
rows total: 2000
time total: 109.462 ms
time per row: 0.054731 ms
I20240502 01:58:02.542237 28136 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52898:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:02.548144 28391 tablet_service.cc:1467] Processing CreateTablet for tablet f2518b866b1743e28cf63dfe56dbdf3e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b20aef5e816a4e81833f4db2d2953e0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:02.548241 28261 tablet_service.cc:1467] Processing CreateTablet for tablet f2518b866b1743e28cf63dfe56dbdf3e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b20aef5e816a4e81833f4db2d2953e0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:02.548499 28391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2518b866b1743e28cf63dfe56dbdf3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:02.548588 28261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2518b866b1743e28cf63dfe56dbdf3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:02.548856 28653 tablet_service.cc:1467] Processing CreateTablet for tablet bb71470fc7c54c0490a156fd4256ff48 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b20aef5e816a4e81833f4db2d2953e0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:02.548879 28654 tablet_service.cc:1467] Processing CreateTablet for tablet f2518b866b1743e28cf63dfe56dbdf3e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b20aef5e816a4e81833f4db2d2953e0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:02.549121 28653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb71470fc7c54c0490a156fd4256ff48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:02.549351 28652 tablet_service.cc:1467] Processing CreateTablet for tablet fa11ebedbf5c4038a5388d9440688011 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b20aef5e816a4e81833f4db2d2953e0c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:02.549585 28652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa11ebedbf5c4038a5388d9440688011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:02.550951 29427 tablet_bootstrap.cc:492] T f2518b866b1743e28cf63dfe56dbdf3e P 8f4edf52bb2d4624ae596b032e50c2a4: Bootstrap starting.
I20240502 01:58:02.551132 29428 tablet_bootstrap.cc:492] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:02.551887 29427 tablet_bootstrap.cc:654] T f2518b866b1743e28cf63dfe56dbdf3e P 8f4edf52bb2d4624ae596b032e50c2a4: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:02.551991 29428 tablet_bootstrap.cc:654] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:02.552243 28654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2518b866b1743e28cf63dfe56dbdf3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:02.552181 28522 tablet_service.cc:1467] Processing CreateTablet for tablet bb71470fc7c54c0490a156fd4256ff48 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b20aef5e816a4e81833f4db2d2953e0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:02.552476 28522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb71470fc7c54c0490a156fd4256ff48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:02.552467 29429 tablet_bootstrap.cc:492] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2: Bootstrap starting.
I20240502 01:58:02.553050 29428 tablet_bootstrap.cc:492] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52: No bootstrap required, opened a new log
I20240502 01:58:02.553126 29428 ts_tablet_manager.cc:1367] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:02.553246 29429 tablet_bootstrap.cc:654] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:02.553588 29428 raft_consensus.cc:348] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.553700 29428 raft_consensus.cc:374] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:02.553731 29428 raft_consensus.cc:729] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:02.553889 29428 consensus_queue.cc:260] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.554194 29428 ts_tablet_manager.cc:1398] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:02.554378 29429 tablet_bootstrap.cc:492] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2: No bootstrap required, opened a new log
I20240502 01:58:02.554509 29429 ts_tablet_manager.cc:1367] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:02.554973 29429 raft_consensus.cc:348] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.555078 29429 raft_consensus.cc:374] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:02.555110 29429 raft_consensus.cc:729] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Initialized, Role: FOLLOWER
I20240502 01:58:02.555157 29433 tablet_bootstrap.cc:492] T bb71470fc7c54c0490a156fd4256ff48 P ef3bc5c942d04bd29540a80415308353: Bootstrap starting.
I20240502 01:58:02.555223 29429 consensus_queue.cc:260] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.555464 29429 ts_tablet_manager.cc:1398] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:02.555428 28261 tablet_service.cc:1467] Processing CreateTablet for tablet bb71470fc7c54c0490a156fd4256ff48 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b20aef5e816a4e81833f4db2d2953e0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:02.555640 29429 tablet_bootstrap.cc:492] T fa11ebedbf5c4038a5388d9440688011 P 8cb8626af432419c922ceefff85f69f2: Bootstrap starting.
I20240502 01:58:02.555651 28261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb71470fc7c54c0490a156fd4256ff48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:02.556087 29433 tablet_bootstrap.cc:654] T bb71470fc7c54c0490a156fd4256ff48 P ef3bc5c942d04bd29540a80415308353: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:02.556483 29429 tablet_bootstrap.cc:654] T fa11ebedbf5c4038a5388d9440688011 P 8cb8626af432419c922ceefff85f69f2: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:02.556967 29427 tablet_bootstrap.cc:492] T f2518b866b1743e28cf63dfe56dbdf3e P 8f4edf52bb2d4624ae596b032e50c2a4: No bootstrap required, opened a new log
I20240502 01:58:02.557052 29427 ts_tablet_manager.cc:1367] T f2518b866b1743e28cf63dfe56dbdf3e P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:58:02.557235 28391 tablet_service.cc:1467] Processing CreateTablet for tablet fa11ebedbf5c4038a5388d9440688011 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b20aef5e816a4e81833f4db2d2953e0c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:02.557428 28391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa11ebedbf5c4038a5388d9440688011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:02.557566 29427 raft_consensus.cc:348] T f2518b866b1743e28cf63dfe56dbdf3e P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.557678 29427 raft_consensus.cc:374] T f2518b866b1743e28cf63dfe56dbdf3e P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:02.557710 29427 raft_consensus.cc:729] T f2518b866b1743e28cf63dfe56dbdf3e P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Initialized, Role: FOLLOWER
I20240502 01:58:02.557854 28260 tablet_service.cc:1467] Processing CreateTablet for tablet fa11ebedbf5c4038a5388d9440688011 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b20aef5e816a4e81833f4db2d2953e0c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:02.557839 29427 consensus_queue.cc:260] T f2518b866b1743e28cf63dfe56dbdf3e P 8f4edf52bb2d4624ae596b032e50c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.558076 28260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa11ebedbf5c4038a5388d9440688011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:02.558144 29427 ts_tablet_manager.cc:1398] T f2518b866b1743e28cf63dfe56dbdf3e P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:02.560544 29427 tablet_bootstrap.cc:492] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4: Bootstrap starting.
I20240502 01:58:02.561161 29428 tablet_bootstrap.cc:492] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:02.561473 29427 tablet_bootstrap.cc:654] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:02.562145 29428 tablet_bootstrap.cc:654] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:02.562441 29429 tablet_bootstrap.cc:492] T fa11ebedbf5c4038a5388d9440688011 P 8cb8626af432419c922ceefff85f69f2: No bootstrap required, opened a new log
I20240502 01:58:02.562541 29429 ts_tablet_manager.cc:1367] T fa11ebedbf5c4038a5388d9440688011 P 8cb8626af432419c922ceefff85f69f2: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240502 01:58:02.563066 29429 raft_consensus.cc:348] T fa11ebedbf5c4038a5388d9440688011 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:02.563191 29429 raft_consensus.cc:374] T fa11ebedbf5c4038a5388d9440688011 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:02.563225 29429 raft_consensus.cc:729] T fa11ebedbf5c4038a5388d9440688011 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Initialized, Role: FOLLOWER
I20240502 01:58:02.563365 29428 tablet_bootstrap.cc:492] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52: No bootstrap required, opened a new log
I20240502 01:58:02.563380 29429 consensus_queue.cc:260] T fa11ebedbf5c4038a5388d9440688011 P 8cb8626af432419c922ceefff85f69f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:02.563469 29428 ts_tablet_manager.cc:1367] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:02.563643 29433 tablet_bootstrap.cc:492] T bb71470fc7c54c0490a156fd4256ff48 P ef3bc5c942d04bd29540a80415308353: No bootstrap required, opened a new log
I20240502 01:58:02.563750 29433 ts_tablet_manager.cc:1367] T bb71470fc7c54c0490a156fd4256ff48 P ef3bc5c942d04bd29540a80415308353: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:58:02.563805 29429 ts_tablet_manager.cc:1398] T fa11ebedbf5c4038a5388d9440688011 P 8cb8626af432419c922ceefff85f69f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:02.563969 29429 tablet_bootstrap.cc:492] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2: Bootstrap starting.
I20240502 01:58:02.564108 29428 raft_consensus.cc:348] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.564199 29428 raft_consensus.cc:374] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:02.564229 29428 raft_consensus.cc:729] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:02.564235 29433 raft_consensus.cc:348] T bb71470fc7c54c0490a156fd4256ff48 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.564355 29428 consensus_queue.cc:260] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.564479 29433 raft_consensus.cc:374] T bb71470fc7c54c0490a156fd4256ff48 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:02.564540 29433 raft_consensus.cc:729] T bb71470fc7c54c0490a156fd4256ff48 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Initialized, Role: FOLLOWER
I20240502 01:58:02.564649 29428 ts_tablet_manager.cc:1398] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:02.564735 29433 consensus_queue.cc:260] T bb71470fc7c54c0490a156fd4256ff48 P ef3bc5c942d04bd29540a80415308353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.564831 29428 tablet_bootstrap.cc:492] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:02.564888 29429 tablet_bootstrap.cc:654] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:02.565057 29433 ts_tablet_manager.cc:1398] T bb71470fc7c54c0490a156fd4256ff48 P ef3bc5c942d04bd29540a80415308353: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:02.565809 29428 tablet_bootstrap.cc:654] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:02.566128 29427 tablet_bootstrap.cc:492] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4: No bootstrap required, opened a new log
I20240502 01:58:02.566224 29427 ts_tablet_manager.cc:1367] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:02.566720 29427 raft_consensus.cc:348] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:02.566828 29427 raft_consensus.cc:374] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:02.566862 29427 raft_consensus.cc:729] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Initialized, Role: FOLLOWER
I20240502 01:58:02.567018 29427 consensus_queue.cc:260] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:02.567307 29427 ts_tablet_manager.cc:1398] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:02.568209 29429 tablet_bootstrap.cc:492] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2: No bootstrap required, opened a new log
I20240502 01:58:02.568305 29429 ts_tablet_manager.cc:1367] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:58:02.568791 29429 raft_consensus.cc:348] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.568897 29429 raft_consensus.cc:374] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:02.568938 29429 raft_consensus.cc:729] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Initialized, Role: FOLLOWER
I20240502 01:58:02.568954 29428 tablet_bootstrap.cc:492] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52: No bootstrap required, opened a new log
I20240502 01:58:02.569041 29428 ts_tablet_manager.cc:1367] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:02.569056 29429 consensus_queue.cc:260] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.569281 29429 ts_tablet_manager.cc:1398] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:02.569527 29428 raft_consensus.cc:348] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:02.569636 29428 raft_consensus.cc:374] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:02.569671 29428 raft_consensus.cc:729] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:02.569798 29428 consensus_queue.cc:260] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:02.570050 29428 ts_tablet_manager.cc:1398] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:02.610668 29288 raft_consensus.cc:482] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:02.610803 29288 raft_consensus.cc:504] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.611256 29288 leader_election.cc:288] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973), ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695)
I20240502 01:58:02.611701 28281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb71470fc7c54c0490a156fd4256ff48" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52" is_pre_election: true
I20240502 01:58:02.611739 28542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb71470fc7c54c0490a156fd4256ff48" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353" is_pre_election: true
I20240502 01:58:02.611881 28281 raft_consensus.cc:2454] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 0.
I20240502 01:58:02.611888 28542 raft_consensus.cc:2454] T bb71470fc7c54c0490a156fd4256ff48 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 0.
I20240502 01:58:02.612236 28609 leader_election.cc:302] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:02.612411 29288 raft_consensus.cc:2781] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:02.612468 29288 raft_consensus.cc:482] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:02.612496 29288 raft_consensus.cc:3037] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.613354 29288 raft_consensus.cc:504] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.613654 29288 leader_election.cc:288] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 election: Requested vote from peers a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973), ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695)
I20240502 01:58:02.614003 28281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb71470fc7c54c0490a156fd4256ff48" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52"
I20240502 01:58:02.614024 28542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb71470fc7c54c0490a156fd4256ff48" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353"
I20240502 01:58:02.614122 28542 raft_consensus.cc:3037] T bb71470fc7c54c0490a156fd4256ff48 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.614130 28281 raft_consensus.cc:3037] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.615025 28542 raft_consensus.cc:2454] T bb71470fc7c54c0490a156fd4256ff48 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 1.
I20240502 01:58:02.615180 28281 raft_consensus.cc:2454] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 1.
I20240502 01:58:02.615321 28609 leader_election.cc:302] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:02.615550 29288 raft_consensus.cc:2781] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:02.615746 29288 raft_consensus.cc:686] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [term 1 LEADER]: Becoming Leader. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Running, Role: LEADER
I20240502 01:58:02.615887 29288 consensus_queue.cc:237] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.617591 28131 catalog_manager.cc:5484] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cb8626af432419c922ceefff85f69f2 (127.27.111.196). New cstate: current_term: 1 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:02.710172 29304 raft_consensus.cc:482] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:02.710326 29304 raft_consensus.cc:504] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:02.710860 29304 leader_election.cc:288] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797), a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:58:02.711427 28281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa11ebedbf5c4038a5388d9440688011" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52" is_pre_election: true
I20240502 01:58:02.711433 28674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa11ebedbf5c4038a5388d9440688011" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2" is_pre_election: true
I20240502 01:58:02.711582 28281 raft_consensus.cc:2454] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 0.
I20240502 01:58:02.711586 28674 raft_consensus.cc:2454] T fa11ebedbf5c4038a5388d9440688011 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 0.
I20240502 01:58:02.711882 28348 leader_election.cc:302] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, 8f4edf52bb2d4624ae596b032e50c2a4; no voters:
I20240502 01:58:02.712091 29304 raft_consensus.cc:2781] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:02.712167 29304 raft_consensus.cc:482] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:02.712208 29304 raft_consensus.cc:3037] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.713002 29304 raft_consensus.cc:504] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:02.713321 29304 leader_election.cc:288] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 election: Requested vote from peers 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797), a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:58:02.713655 28281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa11ebedbf5c4038a5388d9440688011" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52"
I20240502 01:58:02.713661 28674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa11ebedbf5c4038a5388d9440688011" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2"
I20240502 01:58:02.713754 28674 raft_consensus.cc:3037] T fa11ebedbf5c4038a5388d9440688011 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.713764 28281 raft_consensus.cc:3037] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.714561 28281 raft_consensus.cc:2454] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 1.
I20240502 01:58:02.714565 28674 raft_consensus.cc:2454] T fa11ebedbf5c4038a5388d9440688011 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 1.
I20240502 01:58:02.714855 28348 leader_election.cc:302] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, 8f4edf52bb2d4624ae596b032e50c2a4; no voters:
I20240502 01:58:02.715029 29304 raft_consensus.cc:2781] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:02.715119 29304 raft_consensus.cc:686] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Running, Role: LEADER
I20240502 01:58:02.715269 29304 consensus_queue.cc:237] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:02.716981 28131 catalog_manager.cc:5484] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:02.795630 29288 raft_consensus.cc:482] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:02.795760 29288 raft_consensus.cc:504] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.796219 29288 leader_election.cc:288] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369), a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:58:02.796705 28281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2518b866b1743e28cf63dfe56dbdf3e" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52" is_pre_election: true
I20240502 01:58:02.796710 28412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2518b866b1743e28cf63dfe56dbdf3e" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" is_pre_election: true
I20240502 01:58:02.796846 28281 raft_consensus.cc:2454] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 0.
I20240502 01:58:02.796856 28412 raft_consensus.cc:2454] T f2518b866b1743e28cf63dfe56dbdf3e P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 0.
I20240502 01:58:02.797185 28607 leader_election.cc:302] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, 8f4edf52bb2d4624ae596b032e50c2a4; no voters:
I20240502 01:58:02.797396 29288 raft_consensus.cc:2781] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:02.797454 29288 raft_consensus.cc:482] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:02.797505 29288 raft_consensus.cc:3037] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.798431 29288 raft_consensus.cc:504] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.798730 29288 leader_election.cc:288] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 election: Requested vote from peers 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369), a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:58:02.799131 28412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2518b866b1743e28cf63dfe56dbdf3e" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
I20240502 01:58:02.799185 28281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2518b866b1743e28cf63dfe56dbdf3e" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52"
I20240502 01:58:02.799265 28412 raft_consensus.cc:3037] T f2518b866b1743e28cf63dfe56dbdf3e P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.799281 28281 raft_consensus.cc:3037] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:02.800205 28281 raft_consensus.cc:2454] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 1.
I20240502 01:58:02.800205 28412 raft_consensus.cc:2454] T f2518b866b1743e28cf63dfe56dbdf3e P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 1.
I20240502 01:58:02.800477 28607 leader_election.cc:302] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, a962820b582e4e158a2c765c544b8e52; no voters:
I20240502 01:58:02.800628 29288 raft_consensus.cc:2781] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:02.800730 29288 raft_consensus.cc:686] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [term 1 LEADER]: Becoming Leader. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Running, Role: LEADER
I20240502 01:58:02.800870 29288 consensus_queue.cc:237] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:02.802479 28131 catalog_manager.cc:5484] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cb8626af432419c922ceefff85f69f2 (127.27.111.196). New cstate: current_term: 1 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } }
W20240502 01:58:02.963599 29444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:02.963863 29444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:02.966980 29444 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 01:58:03.057799 28281 raft_consensus.cc:1261] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:03.057814 28674 raft_consensus.cc:1261] T fa11ebedbf5c4038a5388d9440688011 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:03.058331 29284 consensus_queue.cc:1035] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:03.058609 29304 consensus_queue.cc:1035] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:03.059630 28280 raft_consensus.cc:1261] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:03.060181 29440 consensus_queue.cc:1035] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:03.060654 28412 raft_consensus.cc:1261] T f2518b866b1743e28cf63dfe56dbdf3e P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:03.061455 28542 raft_consensus.cc:1261] T bb71470fc7c54c0490a156fd4256ff48 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:03.061858 29440 consensus_queue.cc:1035] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:03.062178 29440 consensus_queue.cc:1035] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:03.062317 28279 raft_consensus.cc:1261] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:03.064015 29288 consensus_queue.cc:1035] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.549 ms
time per row: 0.0592743 ms
I20240502 01:58:03.128182 28131 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52898:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:03.133884 28522 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4edc183396406f81e997e8bf6f53fb (DEFAULT_TABLE table=rebalance_test_table_3 [id=86cf304e1bdb4ceabae33570f6c0d7fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.134037 28391 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4edc183396406f81e997e8bf6f53fb (DEFAULT_TABLE table=rebalance_test_table_3 [id=86cf304e1bdb4ceabae33570f6c0d7fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.134239 28522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4edc183396406f81e997e8bf6f53fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.134346 28391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4edc183396406f81e997e8bf6f53fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.134660 28260 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4edc183396406f81e997e8bf6f53fb (DEFAULT_TABLE table=rebalance_test_table_3 [id=86cf304e1bdb4ceabae33570f6c0d7fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.134735 28392 tablet_service.cc:1467] Processing CreateTablet for tablet 30179e5d45eb47cd88045b81e4872c78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=86cf304e1bdb4ceabae33570f6c0d7fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.134927 28260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4edc183396406f81e997e8bf6f53fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.134953 28392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30179e5d45eb47cd88045b81e4872c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.137109 29501 tablet_bootstrap.cc:492] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4: Bootstrap starting.
I20240502 01:58:03.137181 29502 tablet_bootstrap.cc:492] T 1c4edc183396406f81e997e8bf6f53fb P ef3bc5c942d04bd29540a80415308353: Bootstrap starting.
I20240502 01:58:03.137444 28261 tablet_service.cc:1467] Processing CreateTablet for tablet 4f816e20fed94a0587d9c7cfac7071cc (DEFAULT_TABLE table=rebalance_test_table_3 [id=86cf304e1bdb4ceabae33570f6c0d7fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.137703 28261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f816e20fed94a0587d9c7cfac7071cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.138140 29502 tablet_bootstrap.cc:654] T 1c4edc183396406f81e997e8bf6f53fb P ef3bc5c942d04bd29540a80415308353: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.138337 29501 tablet_bootstrap.cc:654] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.138937 28522 tablet_service.cc:1467] Processing CreateTablet for tablet 4f816e20fed94a0587d9c7cfac7071cc (DEFAULT_TABLE table=rebalance_test_table_3 [id=86cf304e1bdb4ceabae33570f6c0d7fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.139021 28523 tablet_service.cc:1467] Processing CreateTablet for tablet 30179e5d45eb47cd88045b81e4872c78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=86cf304e1bdb4ceabae33570f6c0d7fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.139181 28522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f816e20fed94a0587d9c7cfac7071cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.139462 29501 tablet_bootstrap.cc:492] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4: No bootstrap required, opened a new log
I20240502 01:58:03.139482 28523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30179e5d45eb47cd88045b81e4872c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.139478 28654 tablet_service.cc:1467] Processing CreateTablet for tablet 4f816e20fed94a0587d9c7cfac7071cc (DEFAULT_TABLE table=rebalance_test_table_3 [id=86cf304e1bdb4ceabae33570f6c0d7fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.139537 29501 ts_tablet_manager.cc:1367] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:03.139739 28654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f816e20fed94a0587d9c7cfac7071cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.139976 29501 raft_consensus.cc:348] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.140094 29501 raft_consensus.cc:374] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.140126 29501 raft_consensus.cc:729] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.140260 29501 consensus_queue.cc:260] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.140539 29501 ts_tablet_manager.cc:1398] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:03.140712 29501 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4: Bootstrap starting.
I20240502 01:58:03.141575 29501 tablet_bootstrap.cc:654] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.142061 28652 tablet_service.cc:1467] Processing CreateTablet for tablet 30179e5d45eb47cd88045b81e4872c78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=86cf304e1bdb4ceabae33570f6c0d7fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.142284 28652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30179e5d45eb47cd88045b81e4872c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.142452 29502 tablet_bootstrap.cc:492] T 1c4edc183396406f81e997e8bf6f53fb P ef3bc5c942d04bd29540a80415308353: No bootstrap required, opened a new log
I20240502 01:58:03.142529 29502 ts_tablet_manager.cc:1367] T 1c4edc183396406f81e997e8bf6f53fb P ef3bc5c942d04bd29540a80415308353: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:03.142726 29501 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4: No bootstrap required, opened a new log
I20240502 01:58:03.142805 29501 ts_tablet_manager.cc:1367] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:03.142897 29502 raft_consensus.cc:348] T 1c4edc183396406f81e997e8bf6f53fb P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.142992 29502 raft_consensus.cc:374] T 1c4edc183396406f81e997e8bf6f53fb P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.143019 29502 raft_consensus.cc:729] T 1c4edc183396406f81e997e8bf6f53fb P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.143151 29502 consensus_queue.cc:260] T 1c4edc183396406f81e997e8bf6f53fb P ef3bc5c942d04bd29540a80415308353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.143234 29501 raft_consensus.cc:348] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.143321 29501 raft_consensus.cc:374] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.143355 29501 raft_consensus.cc:729] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.143446 29502 ts_tablet_manager.cc:1398] T 1c4edc183396406f81e997e8bf6f53fb P ef3bc5c942d04bd29540a80415308353: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:03.143471 29501 consensus_queue.cc:260] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.143607 29502 tablet_bootstrap.cc:492] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353: Bootstrap starting.
I20240502 01:58:03.143718 29501 ts_tablet_manager.cc:1398] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:03.144376 29502 tablet_bootstrap.cc:654] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.145099 29503 tablet_bootstrap.cc:492] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:03.145417 29502 tablet_bootstrap.cc:492] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353: No bootstrap required, opened a new log
I20240502 01:58:03.145501 29502 ts_tablet_manager.cc:1367] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:03.145812 29502 raft_consensus.cc:348] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.145905 29502 raft_consensus.cc:374] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.145967 29502 raft_consensus.cc:729] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.146108 29503 tablet_bootstrap.cc:654] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.146111 29502 consensus_queue.cc:260] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.146426 29506 tablet_bootstrap.cc:492] T 4f816e20fed94a0587d9c7cfac7071cc P 8cb8626af432419c922ceefff85f69f2: Bootstrap starting.
I20240502 01:58:03.146471 29502 ts_tablet_manager.cc:1398] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:03.146623 29502 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353: Bootstrap starting.
I20240502 01:58:03.147392 29502 tablet_bootstrap.cc:654] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.147884 29503 tablet_bootstrap.cc:492] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52: No bootstrap required, opened a new log
I20240502 01:58:03.147974 29503 ts_tablet_manager.cc:1367] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:03.148433 29502 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353: No bootstrap required, opened a new log
I20240502 01:58:03.148558 29502 ts_tablet_manager.cc:1367] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:03.148972 29502 raft_consensus.cc:348] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.149132 29502 raft_consensus.cc:374] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.149055 29503 raft_consensus.cc:348] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.149178 29503 raft_consensus.cc:374] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.149225 29502 raft_consensus.cc:729] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.149246 29503 raft_consensus.cc:729] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.149425 29502 consensus_queue.cc:260] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.149411 29503 consensus_queue.cc:260] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.149478 29506 tablet_bootstrap.cc:654] T 4f816e20fed94a0587d9c7cfac7071cc P 8cb8626af432419c922ceefff85f69f2: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.149726 29503 ts_tablet_manager.cc:1398] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:03.149834 29502 ts_tablet_manager.cc:1398] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:03.149994 29503 tablet_bootstrap.cc:492] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:03.151361 29506 tablet_bootstrap.cc:492] T 4f816e20fed94a0587d9c7cfac7071cc P 8cb8626af432419c922ceefff85f69f2: No bootstrap required, opened a new log
I20240502 01:58:03.151405 29503 tablet_bootstrap.cc:654] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.151463 29506 ts_tablet_manager.cc:1367] T 4f816e20fed94a0587d9c7cfac7071cc P 8cb8626af432419c922ceefff85f69f2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:03.152364 29506 raft_consensus.cc:348] T 4f816e20fed94a0587d9c7cfac7071cc P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.152478 29503 tablet_bootstrap.cc:492] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52: No bootstrap required, opened a new log
I20240502 01:58:03.152482 29506 raft_consensus.cc:374] T 4f816e20fed94a0587d9c7cfac7071cc P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.152521 29506 raft_consensus.cc:729] T 4f816e20fed94a0587d9c7cfac7071cc P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.152551 29503 ts_tablet_manager.cc:1367] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:03.152704 29506 consensus_queue.cc:260] T 4f816e20fed94a0587d9c7cfac7071cc P 8cb8626af432419c922ceefff85f69f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.152935 29503 raft_consensus.cc:348] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.153056 29503 raft_consensus.cc:374] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.153107 29503 raft_consensus.cc:729] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.153134 29506 ts_tablet_manager.cc:1398] T 4f816e20fed94a0587d9c7cfac7071cc P 8cb8626af432419c922ceefff85f69f2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:03.153224 29503 consensus_queue.cc:260] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.153503 29506 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2: Bootstrap starting.
I20240502 01:58:03.153512 29503 ts_tablet_manager.cc:1398] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:03.154559 29506 tablet_bootstrap.cc:654] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.155961 29506 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2: No bootstrap required, opened a new log
I20240502 01:58:03.156059 29506 ts_tablet_manager.cc:1367] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:03.156491 29506 raft_consensus.cc:348] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.156576 29506 raft_consensus.cc:374] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.156597 29506 raft_consensus.cc:729] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.156703 29506 consensus_queue.cc:260] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.156953 29506 ts_tablet_manager.cc:1398] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:03.180590 29276 raft_consensus.cc:482] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:03.180723 29276 raft_consensus.cc:504] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.181190 29276 leader_election.cc:288] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973), 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:03.181694 28674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f816e20fed94a0587d9c7cfac7071cc" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2" is_pre_election: true
I20240502 01:58:03.181870 28674 raft_consensus.cc:2454] T 4f816e20fed94a0587d9c7cfac7071cc P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 0.
I20240502 01:58:03.182235 28479 leader_election.cc:302] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:03.182487 29276 raft_consensus.cc:2781] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:03.182543 29276 raft_consensus.cc:482] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:03.182570 29276 raft_consensus.cc:3037] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.183416 29276 raft_consensus.cc:504] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.183743 29276 leader_election.cc:288] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 election: Requested vote from peers a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973), 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:03.184093 28674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f816e20fed94a0587d9c7cfac7071cc" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2"
I20240502 01:58:03.184221 28674 raft_consensus.cc:3037] T 4f816e20fed94a0587d9c7cfac7071cc P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.185037 28674 raft_consensus.cc:2454] T 4f816e20fed94a0587d9c7cfac7071cc P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 1.
I20240502 01:58:03.185356 28479 leader_election.cc:302] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:03.185343 28281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f816e20fed94a0587d9c7cfac7071cc" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52" is_pre_election: true
I20240502 01:58:03.185343 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f816e20fed94a0587d9c7cfac7071cc" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52"
I20240502 01:58:03.185496 28280 raft_consensus.cc:3037] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.185536 29276 raft_consensus.cc:2781] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:03.185741 29276 raft_consensus.cc:686] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Becoming Leader. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Running, Role: LEADER
I20240502 01:58:03.185922 29276 consensus_queue.cc:237] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.186448 28280 raft_consensus.cc:2454] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 1.
I20240502 01:58:03.187729 28132 catalog_manager.cc:5484] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef3bc5c942d04bd29540a80415308353 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "ef3bc5c942d04bd29540a80415308353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:03.208647 29516 raft_consensus.cc:482] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:03.208824 29516 raft_consensus.cc:504] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.209431 29516 leader_election.cc:288] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369), 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:03.209686 28674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2" is_pre_election: true
I20240502 01:58:03.209760 28412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" is_pre_election: true
I20240502 01:58:03.209820 28674 raft_consensus.cc:2454] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 0.
I20240502 01:58:03.209926 28412 raft_consensus.cc:2454] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 0.
I20240502 01:58:03.210151 28479 leader_election.cc:302] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:03.210332 29516 raft_consensus.cc:2781] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:03.210428 29516 raft_consensus.cc:482] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:03.210464 29516 raft_consensus.cc:3037] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.211398 29516 raft_consensus.cc:504] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.211733 29516 leader_election.cc:288] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 election: Requested vote from peers 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369), 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:03.212056 28412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
I20240502 01:58:03.212065 28674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2"
I20240502 01:58:03.212162 28674 raft_consensus.cc:3037] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.212199 28412 raft_consensus.cc:3037] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.212962 28674 raft_consensus.cc:2454] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 1.
I20240502 01:58:03.213032 28412 raft_consensus.cc:2454] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 1.
I20240502 01:58:03.213224 28479 leader_election.cc:302] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:03.213395 29516 raft_consensus.cc:2781] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:03.213482 29516 raft_consensus.cc:686] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Becoming Leader. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Running, Role: LEADER
I20240502 01:58:03.213613 29516 consensus_queue.cc:237] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.215500 28132 catalog_manager.cc:5484] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef3bc5c942d04bd29540a80415308353 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "ef3bc5c942d04bd29540a80415308353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:03.457271 29284 raft_consensus.cc:482] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:03.457453 29284 raft_consensus.cc:504] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.457906 29284 leader_election.cc:288] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695), a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:58:03.458515 28542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4edc183396406f81e997e8bf6f53fb" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353" is_pre_election: true
I20240502 01:58:03.458523 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4edc183396406f81e997e8bf6f53fb" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52" is_pre_election: true
I20240502 01:58:03.458669 28280 raft_consensus.cc:2454] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 0.
I20240502 01:58:03.458668 28542 raft_consensus.cc:2454] T 1c4edc183396406f81e997e8bf6f53fb P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 0.
I20240502 01:58:03.459004 28345 leader_election.cc:302] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f4edf52bb2d4624ae596b032e50c2a4, a962820b582e4e158a2c765c544b8e52; no voters:
I20240502 01:58:03.459178 29284 raft_consensus.cc:2781] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:03.459232 29284 raft_consensus.cc:482] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:03.459275 29284 raft_consensus.cc:3037] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.460146 29284 raft_consensus.cc:504] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.460489 29284 leader_election.cc:288] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 election: Requested vote from peers ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695), a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:58:03.460886 28542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4edc183396406f81e997e8bf6f53fb" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353"
I20240502 01:58:03.460886 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4edc183396406f81e997e8bf6f53fb" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52"
I20240502 01:58:03.460995 28280 raft_consensus.cc:3037] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.461011 28542 raft_consensus.cc:3037] T 1c4edc183396406f81e997e8bf6f53fb P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.461779 28280 raft_consensus.cc:2454] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 1.
I20240502 01:58:03.461802 28542 raft_consensus.cc:2454] T 1c4edc183396406f81e997e8bf6f53fb P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 1.
I20240502 01:58:03.462085 28345 leader_election.cc:302] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f4edf52bb2d4624ae596b032e50c2a4, a962820b582e4e158a2c765c544b8e52; no voters:
I20240502 01:58:03.462256 29284 raft_consensus.cc:2781] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:03.462349 29284 raft_consensus.cc:686] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Becoming Leader. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Running, Role: LEADER
I20240502 01:58:03.462517 29284 consensus_queue.cc:237] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.464411 28132 catalog_manager.cc:5484] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:03.592312 29516 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:03.597896 29516 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:03.631780 29520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:03.632082 29520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:03.635313 29520 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 01:58:03.693982 29516 consensus_queue.cc:1035] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:03.700522 29276 consensus_queue.cc:1035] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:03.729539 28542 raft_consensus.cc:1261] T 1c4edc183396406f81e997e8bf6f53fb P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:03.729573 28280 raft_consensus.cc:1261] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:03.730008 29304 consensus_queue.cc:1035] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:03.730149 29284 consensus_queue.cc:1035] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 126.282 ms
time per row: 0.0631408 ms
I20240502 01:58:03.805760 28130 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52898:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:03.811439 28261 tablet_service.cc:1467] Processing CreateTablet for tablet cd3a11e1ac624f3fbc8cacf351b8afa5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6a3a04e2234e47ff9c73333582341e02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.811563 28523 tablet_service.cc:1467] Processing CreateTablet for tablet cd3a11e1ac624f3fbc8cacf351b8afa5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6a3a04e2234e47ff9c73333582341e02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.811803 28523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd3a11e1ac624f3fbc8cacf351b8afa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.811805 28261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd3a11e1ac624f3fbc8cacf351b8afa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.812356 28392 tablet_service.cc:1467] Processing CreateTablet for tablet cd3a11e1ac624f3fbc8cacf351b8afa5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6a3a04e2234e47ff9c73333582341e02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.812441 28391 tablet_service.cc:1467] Processing CreateTablet for tablet 14037ba64754436ba114928a3e0dcbe0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6a3a04e2234e47ff9c73333582341e02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.812675 28392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd3a11e1ac624f3fbc8cacf351b8afa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.813159 28259 tablet_service.cc:1467] Processing CreateTablet for tablet cb5e784833984a5a95c1340c96d15994 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6a3a04e2234e47ff9c73333582341e02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.813390 28259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb5e784833984a5a95c1340c96d15994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.814268 28652 tablet_service.cc:1467] Processing CreateTablet for tablet 14037ba64754436ba114928a3e0dcbe0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6a3a04e2234e47ff9c73333582341e02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.814532 28652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14037ba64754436ba114928a3e0dcbe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.815013 29578 tablet_bootstrap.cc:492] T cd3a11e1ac624f3fbc8cacf351b8afa5 P 8f4edf52bb2d4624ae596b032e50c2a4: Bootstrap starting.
I20240502 01:58:03.815264 28391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14037ba64754436ba114928a3e0dcbe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.815593 28260 tablet_service.cc:1467] Processing CreateTablet for tablet 14037ba64754436ba114928a3e0dcbe0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6a3a04e2234e47ff9c73333582341e02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.815801 28260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14037ba64754436ba114928a3e0dcbe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.816001 29578 tablet_bootstrap.cc:654] T cd3a11e1ac624f3fbc8cacf351b8afa5 P 8f4edf52bb2d4624ae596b032e50c2a4: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.817097 28654 tablet_service.cc:1467] Processing CreateTablet for tablet cb5e784833984a5a95c1340c96d15994 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6a3a04e2234e47ff9c73333582341e02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.817359 28654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb5e784833984a5a95c1340c96d15994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.817802 28523 tablet_service.cc:1467] Processing CreateTablet for tablet cb5e784833984a5a95c1340c96d15994 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6a3a04e2234e47ff9c73333582341e02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:03.817970 29576 tablet_bootstrap.cc:492] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:03.818039 28523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb5e784833984a5a95c1340c96d15994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:03.818915 29576 tablet_bootstrap.cc:654] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.819270 29577 tablet_bootstrap.cc:492] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353: Bootstrap starting.
I20240502 01:58:03.817085 29580 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2: Bootstrap starting.
I20240502 01:58:03.820199 29578 tablet_bootstrap.cc:492] T cd3a11e1ac624f3fbc8cacf351b8afa5 P 8f4edf52bb2d4624ae596b032e50c2a4: No bootstrap required, opened a new log
I20240502 01:58:03.820390 29578 ts_tablet_manager.cc:1367] T cd3a11e1ac624f3fbc8cacf351b8afa5 P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:03.820477 29577 tablet_bootstrap.cc:654] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.820946 29580 tablet_bootstrap.cc:654] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.820967 29578 raft_consensus.cc:348] T cd3a11e1ac624f3fbc8cacf351b8afa5 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.821138 29578 raft_consensus.cc:374] T cd3a11e1ac624f3fbc8cacf351b8afa5 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.821211 29578 raft_consensus.cc:729] T cd3a11e1ac624f3fbc8cacf351b8afa5 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.821379 29578 consensus_queue.cc:260] T cd3a11e1ac624f3fbc8cacf351b8afa5 P 8f4edf52bb2d4624ae596b032e50c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.821604 29577 tablet_bootstrap.cc:492] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353: No bootstrap required, opened a new log
I20240502 01:58:03.821688 29577 ts_tablet_manager.cc:1367] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:03.821691 29578 ts_tablet_manager.cc:1398] T cd3a11e1ac624f3fbc8cacf351b8afa5 P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:03.822187 29576 tablet_bootstrap.cc:492] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52: No bootstrap required, opened a new log
I20240502 01:58:03.822180 29577 raft_consensus.cc:348] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.822292 29577 raft_consensus.cc:374] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.822333 29577 raft_consensus.cc:729] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.822304 29576 ts_tablet_manager.cc:1367] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:03.822475 29577 consensus_queue.cc:260] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.822535 29578 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: Bootstrap starting.
I20240502 01:58:03.822718 29580 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2: No bootstrap required, opened a new log
I20240502 01:58:03.822760 29577 ts_tablet_manager.cc:1398] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:03.822793 29580 ts_tablet_manager.cc:1367] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:03.822930 29577 tablet_bootstrap.cc:492] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353: Bootstrap starting.
I20240502 01:58:03.822978 29576 raft_consensus.cc:348] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.823068 29576 raft_consensus.cc:374] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.823098 29576 raft_consensus.cc:729] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.823220 29576 consensus_queue.cc:260] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.823235 29580 raft_consensus.cc:348] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.823336 29580 raft_consensus.cc:374] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.823370 29580 raft_consensus.cc:729] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.823513 29576 ts_tablet_manager.cc:1398] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:03.823491 29580 consensus_queue.cc:260] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.823671 29576 tablet_bootstrap.cc:492] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:03.823760 29580 ts_tablet_manager.cc:1398] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:03.823787 29577 tablet_bootstrap.cc:654] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.823920 29580 tablet_bootstrap.cc:492] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2: Bootstrap starting.
I20240502 01:58:03.824621 29576 tablet_bootstrap.cc:654] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.824769 29580 tablet_bootstrap.cc:654] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.826957 29576 tablet_bootstrap.cc:492] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52: No bootstrap required, opened a new log
I20240502 01:58:03.827049 29577 tablet_bootstrap.cc:492] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353: No bootstrap required, opened a new log
I20240502 01:58:03.827044 29576 ts_tablet_manager.cc:1367] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:03.827116 29577 ts_tablet_manager.cc:1367] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:03.827530 29576 raft_consensus.cc:348] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:03.827648 29576 raft_consensus.cc:374] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.827690 29576 raft_consensus.cc:729] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.827826 29576 consensus_queue.cc:260] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:03.828100 29576 ts_tablet_manager.cc:1398] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:03.828258 29576 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:03.828387 29577 raft_consensus.cc:348] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:03.828478 29577 raft_consensus.cc:374] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.828749 29577 raft_consensus.cc:729] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.828922 29577 consensus_queue.cc:260] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:03.829164 29580 tablet_bootstrap.cc:492] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2: No bootstrap required, opened a new log
I20240502 01:58:03.829206 29576 tablet_bootstrap.cc:654] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.829245 29580 ts_tablet_manager.cc:1367] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 01:58:03.829718 29580 raft_consensus.cc:348] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:03.829818 29580 raft_consensus.cc:374] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.829859 29580 raft_consensus.cc:729] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.829962 29580 consensus_queue.cc:260] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:03.830169 29577 ts_tablet_manager.cc:1398] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:03.830255 29580 ts_tablet_manager.cc:1398] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:03.830474 29578 tablet_bootstrap.cc:654] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:03.831153 29576 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52: No bootstrap required, opened a new log
I20240502 01:58:03.831266 29576 ts_tablet_manager.cc:1367] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:03.831887 29576 raft_consensus.cc:348] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.832629 29576 raft_consensus.cc:374] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.832660 29578 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: No bootstrap required, opened a new log
I20240502 01:58:03.832718 29576 raft_consensus.cc:729] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.832751 29578 ts_tablet_manager.cc:1367] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240502 01:58:03.832918 29576 consensus_queue.cc:260] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.833294 29576 ts_tablet_manager.cc:1398] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:03.833261 29578 raft_consensus.cc:348] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.833366 29578 raft_consensus.cc:374] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:03.833433 29578 raft_consensus.cc:729] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Initialized, Role: FOLLOWER
I20240502 01:58:03.833660 29578 consensus_queue.cc:260] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:03.833979 29578 ts_tablet_manager.cc:1398] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:03.870404 29546 raft_consensus.cc:482] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:03.870551 29546 raft_consensus.cc:504] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:03.871647 29546 leader_election.cc:288] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797), ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695)
I20240502 01:58:03.875312 29568 raft_consensus.cc:482] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:03.875368 28542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5e784833984a5a95c1340c96d15994" candidate_uuid: "a962820b582e4e158a2c765c544b8e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353" is_pre_election: true
I20240502 01:58:03.875438 29568 raft_consensus.cc:504] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.875459 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5e784833984a5a95c1340c96d15994" candidate_uuid: "a962820b582e4e158a2c765c544b8e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2" is_pre_election: true
I20240502 01:58:03.875535 28542 raft_consensus.cc:2454] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a962820b582e4e158a2c765c544b8e52 in term 0.
I20240502 01:58:03.875573 28673 raft_consensus.cc:2454] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a962820b582e4e158a2c765c544b8e52 in term 0.
I20240502 01:58:03.875859 29568 leader_election.cc:288] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973), 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369)
I20240502 01:58:03.875890 28217 leader_election.cc:302] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, a962820b582e4e158a2c765c544b8e52; no voters:
I20240502 01:58:03.876175 29546 raft_consensus.cc:2781] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:03.876242 29546 raft_consensus.cc:482] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:03.876281 29546 raft_consensus.cc:3037] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.876318 28412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3a11e1ac624f3fbc8cacf351b8afa5" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" is_pre_election: true
I20240502 01:58:03.876477 28412 raft_consensus.cc:2454] T cd3a11e1ac624f3fbc8cacf351b8afa5 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 0.
I20240502 01:58:03.876546 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3a11e1ac624f3fbc8cacf351b8afa5" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52" is_pre_election: true
I20240502 01:58:03.876663 28280 raft_consensus.cc:2454] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 0.
I20240502 01:58:03.876780 28476 leader_election.cc:302] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f4edf52bb2d4624ae596b032e50c2a4, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:03.876940 29568 raft_consensus.cc:2781] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:03.876994 29568 raft_consensus.cc:482] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:03.877030 29568 raft_consensus.cc:3037] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.877391 29546 raft_consensus.cc:504] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:03.877699 29546 leader_election.cc:288] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [CANDIDATE]: Term 1 election: Requested vote from peers 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797), ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695)
I20240502 01:58:03.877976 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5e784833984a5a95c1340c96d15994" candidate_uuid: "a962820b582e4e158a2c765c544b8e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2"
I20240502 01:58:03.877983 29568 raft_consensus.cc:504] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.878091 28673 raft_consensus.cc:3037] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.878312 28542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5e784833984a5a95c1340c96d15994" candidate_uuid: "a962820b582e4e158a2c765c544b8e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353"
I20240502 01:58:03.878444 28542 raft_consensus.cc:3037] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.878703 28412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3a11e1ac624f3fbc8cacf351b8afa5" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
I20240502 01:58:03.878769 28280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3a11e1ac624f3fbc8cacf351b8afa5" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a962820b582e4e158a2c765c544b8e52"
I20240502 01:58:03.878809 28412 raft_consensus.cc:3037] T cd3a11e1ac624f3fbc8cacf351b8afa5 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.878863 28280 raft_consensus.cc:3037] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:03.879146 28673 raft_consensus.cc:2454] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a962820b582e4e158a2c765c544b8e52 in term 1.
I20240502 01:58:03.879335 28542 raft_consensus.cc:2454] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a962820b582e4e158a2c765c544b8e52 in term 1.
I20240502 01:58:03.878319 29568 leader_election.cc:288] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 election: Requested vote from peers a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973), 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369)
I20240502 01:58:03.879658 28217 leader_election.cc:302] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, a962820b582e4e158a2c765c544b8e52; no voters:
I20240502 01:58:03.879724 28412 raft_consensus.cc:2454] T cd3a11e1ac624f3fbc8cacf351b8afa5 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 1.
I20240502 01:58:03.879724 28280 raft_consensus.cc:2454] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 1.
I20240502 01:58:03.879832 29546 raft_consensus.cc:2781] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:03.880051 29546 raft_consensus.cc:686] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [term 1 LEADER]: Becoming Leader. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Running, Role: LEADER
I20240502 01:58:03.880085 28476 leader_election.cc:302] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a962820b582e4e158a2c765c544b8e52, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:03.880241 29568 raft_consensus.cc:2781] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:03.880321 29568 raft_consensus.cc:686] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Becoming Leader. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Running, Role: LEADER
I20240502 01:58:03.880429 29568 consensus_queue.cc:237] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:03.880420 29546 consensus_queue.cc:237] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:03.882335 28134 catalog_manager.cc:5484] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef3bc5c942d04bd29540a80415308353 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "ef3bc5c942d04bd29540a80415308353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:03.882930 28130 catalog_manager.cc:5484] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 reported cstate change: term changed from 0 to 1, leader changed from <none> to a962820b582e4e158a2c765c544b8e52 (127.27.111.193). New cstate: current_term: 1 leader_uuid: "a962820b582e4e158a2c765c544b8e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } health_report { overall_health: HEALTHY } } }
I20240502 01:58:04.205937 29546 raft_consensus.cc:482] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:04.206094 29546 raft_consensus.cc:504] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:04.206666 29546 leader_election.cc:288] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369), 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:04.207216 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14037ba64754436ba114928a3e0dcbe0" candidate_uuid: "a962820b582e4e158a2c765c544b8e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2" is_pre_election: true
I20240502 01:58:04.207401 28673 raft_consensus.cc:2454] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a962820b582e4e158a2c765c544b8e52 in term 0.
I20240502 01:58:04.207762 28217 leader_election.cc:302] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, a962820b582e4e158a2c765c544b8e52; no voters:
I20240502 01:58:04.207937 29546 raft_consensus.cc:2781] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:04.207991 29546 raft_consensus.cc:482] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:04.208027 29546 raft_consensus.cc:3037] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:04.209064 29546 raft_consensus.cc:504] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:04.209395 29546 leader_election.cc:288] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [CANDIDATE]: Term 1 election: Requested vote from peers 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369), 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:04.209821 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14037ba64754436ba114928a3e0dcbe0" candidate_uuid: "a962820b582e4e158a2c765c544b8e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2"
I20240502 01:58:04.209985 28673 raft_consensus.cc:3037] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:04.210950 28412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14037ba64754436ba114928a3e0dcbe0" candidate_uuid: "a962820b582e4e158a2c765c544b8e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" is_pre_election: true
I20240502 01:58:04.211037 28673 raft_consensus.cc:2454] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a962820b582e4e158a2c765c544b8e52 in term 1.
I20240502 01:58:04.211006 28411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14037ba64754436ba114928a3e0dcbe0" candidate_uuid: "a962820b582e4e158a2c765c544b8e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
I20240502 01:58:04.211088 28412 raft_consensus.cc:2454] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a962820b582e4e158a2c765c544b8e52 in term 0.
I20240502 01:58:04.211385 28217 leader_election.cc:302] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, a962820b582e4e158a2c765c544b8e52; no voters:
I20240502 01:58:04.211560 29546 raft_consensus.cc:2781] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:04.211652 29546 raft_consensus.cc:686] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 1 LEADER]: Becoming Leader. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Running, Role: LEADER
I20240502 01:58:04.211808 29546 consensus_queue.cc:237] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:04.213898 28130 catalog_manager.cc:5484] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 reported cstate change: term changed from 0 to 1, leader changed from <none> to a962820b582e4e158a2c765c544b8e52 (127.27.111.193). New cstate: current_term: 1 leader_uuid: "a962820b582e4e158a2c765c544b8e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:04.285555 29571 consensus_queue.cc:1035] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:04.289053 29568 consensus_queue.cc:1035] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:04.382339 29546 consensus_queue.cc:1035] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:04.386612 29546 consensus_queue.cc:1035] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:58:04.391268 29597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:04.391552 29597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:04.394636 29597 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 01:58:04.485971 28673 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer a962820b582e4e158a2c765c544b8e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:04.486474 29546 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:04.490149 28412 raft_consensus.cc:3037] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:04.491566 28412 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Refusing update from remote peer a962820b582e4e158a2c765c544b8e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:04.492089 29592 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.473 ms
time per row: 0.0557366 ms
I20240502 01:58:04.549935 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.197:40085
--local_ip_for_outbound_sockets=127.27.111.197
--tserver_master_addrs=127.27.111.254:42667
--webserver_port=35433
--webserver_interface=127.27.111.197
--builtin_ntp_servers=127.27.111.212:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 01:58:04.664587 29660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:04.664805 29660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:04.664878 29660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:04.668035 29660 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:04.668109 29660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:04.668210 29660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.197
I20240502 01:58:04.672119 29660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.197:40085
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.111.197
--webserver_port=35433
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.111.254:42667
--never_fsync=true
--heap_profile_path=/tmp/kudu.29660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.197
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:04.673099 29660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:04.674252 29660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:58:04.683223 29660 server_base.cc:1034] running on GCE node
W20240502 01:58:04.683609 29668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:04.683751 29666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:04.684018 29665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:04.684410 29660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:04.684981 29660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:04.686120 29660 hybrid_clock.cc:648] HybridClock initialized: now 1714615084686099 us; error 39 us; skew 500 ppm
I20240502 01:58:04.688059 29660 webserver.cc:460] Webserver started at http://127.27.111.197:35433/ using document root <none> and password file <none>
I20240502 01:58:04.688630 29660 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:04.688705 29660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:04.692143 29660 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20240502 01:58:04.694061 29674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.695173 29660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:04.695287 29660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a1a0ca3ab1ca4872b2bb4395068b7033"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:58:04.695734 29660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:04.714764 29660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:04.715541 29660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:04.715708 29660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:04.716307 29660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:04.717157 29660 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:04.717202 29660 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.717242 29660 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:04.717262 29660 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.725498 29660 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.197:40085
I20240502 01:58:04.725524 29787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.197:40085 every 8 connection(s)
I20240502 01:58:04.726658 29660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:58:04.735164 29788 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42667
I20240502 01:58:04.735428 29788 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:04.736013 29788 heartbeater.cc:511] Master 127.27.111.254:42667 requested a full tablet report, sending...
I20240502 01:58:04.736027 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 29660
I20240502 01:58:04.736416 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.198:43065
--local_ip_for_outbound_sockets=127.27.111.198
--tserver_master_addrs=127.27.111.254:42667
--webserver_port=41319
--webserver_interface=127.27.111.198
--builtin_ntp_servers=127.27.111.212:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:58:04.737278 28130 ts_manager.cc:194] Re-registered known tserver with Master: a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197:40085)
I20240502 01:58:04.738039 28130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.197:41537
W20240502 01:58:04.849036 29791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:04.849292 29791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:04.849367 29791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:04.852768 29791 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:04.852860 29791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:04.852960 29791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.198
I20240502 01:58:04.856859 29791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.198:43065
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.111.198
--webserver_port=41319
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.111.254:42667
--never_fsync=true
--heap_profile_path=/tmp/kudu.29791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.198
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:04.857865 29791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:04.859056 29791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:04.867425 29797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:04.867357 29796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:04.868098 29799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:04.868824 29791 server_base.cc:1034] running on GCE node
I20240502 01:58:04.869292 29791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:04.869899 29791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:04.871055 29791 hybrid_clock.cc:648] HybridClock initialized: now 1714615084871025 us; error 50 us; skew 500 ppm
I20240502 01:58:04.873114 29791 webserver.cc:460] Webserver started at http://127.27.111.198:41319/ using document root <none> and password file <none>
I20240502 01:58:04.873692 29791 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:04.873773 29791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:04.877221 29791 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:04.879196 29805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.880219 29791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:04.880335 29791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "34a8610adef8401085b62d8fb1fbc8fe"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-0pw1"
I20240502 01:58:04.880779 29791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:04.895516 29791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:04.896173 29791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:04.896355 29791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:04.896934 29791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:04.897936 29791 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:04.897998 29791 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.898056 29791 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:04.898087 29791 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:04.906733 29918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.198:43065 every 8 connection(s)
I20240502 01:58:04.906713 29791 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.198:43065
I20240502 01:58:04.907897 29791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:58:04.913030 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 29791
I20240502 01:58:04.913429 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.199:42203
--local_ip_for_outbound_sockets=127.27.111.199
--tserver_master_addrs=127.27.111.254:42667
--webserver_port=44385
--webserver_interface=127.27.111.199
--builtin_ntp_servers=127.27.111.212:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:58:04.919060 29919 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42667
I20240502 01:58:04.919337 29919 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:04.919894 29919 heartbeater.cc:511] Master 127.27.111.254:42667 requested a full tablet report, sending...
I20240502 01:58:04.920992 28130 ts_manager.cc:194] Re-registered known tserver with Master: 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198:43065)
I20240502 01:58:04.921561 28130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.198:49425
W20240502 01:58:05.028653 29922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:05.028906 29922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:05.028988 29922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:05.032384 29922 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:05.032476 29922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:05.032580 29922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.199
I20240502 01:58:05.036573 29922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.199:42203
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.111.199
--webserver_port=44385
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.111.254:42667
--never_fsync=true
--heap_profile_path=/tmp/kudu.29922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.199
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:05.037639 29922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:05.038894 29922 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:05.047116 29927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:05.047478 29928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:05.047652 29930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:05.048257 29922 server_base.cc:1034] running on GCE node
I20240502 01:58:05.048638 29922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:05.049207 29922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:05.050364 29922 hybrid_clock.cc:648] HybridClock initialized: now 1714615085050346 us; error 35 us; skew 500 ppm
I20240502 01:58:05.052345 29922 webserver.cc:460] Webserver started at http://127.27.111.199:44385/ using document root <none> and password file <none>
I20240502 01:58:05.052917 29922 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:05.053005 29922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:05.056448 29922 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:58:05.058317 29936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.059356 29922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:05.059465 29922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "dd5f234ed11745168cd67ee403ff098e"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-0pw1"
I20240502 01:58:05.059885 29922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:05.078805 29922 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:05.079516 29922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:05.079689 29922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:05.080257 29922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:05.081194 29922 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:05.081244 29922 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.081301 29922 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:05.081334 29922 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.090996 29922 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.199:42203
I20240502 01:58:05.091015 30049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.199:42203 every 8 connection(s)
I20240502 01:58:05.092105 29922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:58:05.100279 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 29922
I20240502 01:58:05.100847 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.200:39123
--local_ip_for_outbound_sockets=127.27.111.200
--tserver_master_addrs=127.27.111.254:42667
--webserver_port=39877
--webserver_interface=127.27.111.200
--builtin_ntp_servers=127.27.111.212:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:58:05.101658 30050 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42667
I20240502 01:58:05.101931 30050 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:05.102641 30050 heartbeater.cc:511] Master 127.27.111.254:42667 requested a full tablet report, sending...
I20240502 01:58:05.103749 28130 ts_manager.cc:194] Re-registered known tserver with Master: dd5f234ed11745168cd67ee403ff098e (127.27.111.199:42203)
I20240502 01:58:05.104573 28130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.199:54565
W20240502 01:58:05.218032 30053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:05.218340 30053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:05.218437 30053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:05.222877 30053 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:05.222970 30053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:05.223075 30053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.200
I20240502 01:58:05.228242 30053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.200:39123
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.111.200
--webserver_port=39877
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.111.254:42667
--never_fsync=true
--heap_profile_path=/tmp/kudu.30053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.200
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:05.229408 30053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:05.230832 30053 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:05.240244 30058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:05.240849 30061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:05.241003 30059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:05.241881 30053 server_base.cc:1034] running on GCE node
I20240502 01:58:05.242328 30053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:05.242987 30053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:05.244123 30053 hybrid_clock.cc:648] HybridClock initialized: now 1714615085244101 us; error 42 us; skew 500 ppm
I20240502 01:58:05.253784 30053 webserver.cc:460] Webserver started at http://127.27.111.200:39877/ using document root <none> and password file <none>
I20240502 01:58:05.254487 30053 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:05.254566 30053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:05.258010 30053 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:58:05.260078 30067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.261229 30053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:05.261372 30053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0773aa6e61f442bb86d966ae9e97cbc6"
format_stamp: "Formatted at 2024-05-02 01:57:57 on dist-test-slave-0pw1"
I20240502 01:58:05.261826 30053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:05.287030 30053 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:05.287889 30053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:05.288064 30053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:05.289158 30053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:05.290319 30053 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:05.290376 30053 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.290452 30053 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:05.290485 30053 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:05.301800 30053 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.200:39123
I20240502 01:58:05.301854 30180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.200:39123 every 8 connection(s)
I20240502 01:58:05.303138 30053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:58:05.308809 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 30053
I20240502 01:58:05.315457 30181 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42667
I20240502 01:58:05.315757 30181 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:05.316521 30181 heartbeater.cc:511] Master 127.27.111.254:42667 requested a full tablet report, sending...
I20240502 01:58:05.317848 28130 ts_manager.cc:194] Re-registered known tserver with Master: 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200:39123)
I20240502 01:58:05.318727 28130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.200:58379
I20240502 01:58:05.528178 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:05.529945 30115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:05.536048 28391 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:05.536938 28260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:05.540321 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:05.543530 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:05.546487 29722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:05.554100 28523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:05.740590 29788 heartbeater.cc:503] Master 127.27.111.254:42667 was elected leader, sending a full tablet report...
I20240502 01:58:05.748787 28136 ts_manager.cc:292] Set tserver state for a962820b582e4e158a2c765c544b8e52 to MAINTENANCE_MODE
W20240502 01:58:05.871985 30245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:05.872242 30245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:05.875408 30245 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:58:05.923615 29919 heartbeater.cc:503] Master 127.27.111.254:42667 was elected leader, sending a full tablet report...
I20240502 01:58:05.970319 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:05.973261 30115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:05.975749 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:05.983126 28391 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:05.994915 29722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.005678 28523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:06.008188 28260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:06.014113 30115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.020171 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.024189 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:06.036870 28391 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:06.040125 29722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.067144 28523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:06.072296 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.076819 28260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:06.082463 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.106639 30050 heartbeater.cc:503] Master 127.27.111.254:42667 was elected leader, sending a full tablet report...
I20240502 01:58:06.208561 30115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.210721 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:06.223484 28260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:06.224812 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.235041 28523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:06.236696 28391 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:06.252216 30321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:06.252534 30321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:06.257261 30321 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:58:06.260254 29722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.262607 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.321049 30181 heartbeater.cc:503] Master 127.27.111.254:42667 was elected leader, sending a full tablet report...
I20240502 01:58:06.358523 30115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.360687 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.364982 28523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:06.371404 28260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:06.372341 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.377049 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:06.387823 28391 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:06.402666 29722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.439036 29722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.439719 30115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.444808 28260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:06.454418 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.456146 28391 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:06.456614 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:06.471015 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.486869 28523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:06.520284 28281 consensus_queue.cc:237] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:06.521692 28412 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Refusing update from remote peer a962820b582e4e158a2c765c544b8e52: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:06.522106 29650 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:06.522264 28673 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer a962820b582e4e158a2c765c544b8e52: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:06.523593 29650 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:06.525545 29650 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } }
I20240502 01:58:06.525852 28673 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } }
W20240502 01:58:06.622871 30438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:06.623112 30438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:58:06.627476 28130 catalog_manager.cc:5484] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a962820b582e4e158a2c765c544b8e52" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } }
W20240502 01:58:06.627686 30438 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:58:06.628911 28412 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } }
I20240502 01:58:06.632830 28281 consensus_queue.cc:237] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:06.634904 28673 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer a962820b582e4e158a2c765c544b8e52: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:06.635316 29650 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:06.637032 29650 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } } }
I20240502 01:58:06.638722 28673 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } } }
W20240502 01:58:06.639580 28214 consensus_peers.cc:485] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 -> Peer 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198:43065): Couldn't send request to peer 34a8610adef8401085b62d8fb1fbc8fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14037ba64754436ba114928a3e0dcbe0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:06.712101 30115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.712101 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.720571 29722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.721809 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.729220 28411 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Refusing update from remote peer a962820b582e4e158a2c765c544b8e52: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:06.731346 28122 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 14037ba64754436ba114928a3e0dcbe0 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:58:06.731611 28134 catalog_manager.cc:5484] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 reported cstate change: config changed from index 4 to 5, NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "a962820b582e4e158a2c765c544b8e52" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:06.733741 30445 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:06.735652 28411 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } } }
I20240502 01:58:06.738525 28391 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:06.746948 28260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:06.748545 28523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:06.753952 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Runtime error: ksck discovered errors
W20240502 01:58:06.988610 30494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:06.988888 30494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:06.992059 30494 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:58:07.072788 28391 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:07.072782 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.078264 30115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.078543 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:07.083887 29722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.100744 28523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:07.110337 28260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:07.115056 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.125543 30537 ts_tablet_manager.cc:902] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe: Initiating tablet copy from peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:58:07.126636 30537 tablet_copy_client.cc:308] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:34973
I20240502 01:58:07.131736 28301 tablet_copy_service.cc:140] P a962820b582e4e158a2c765c544b8e52: Received BeginTabletCopySession request for tablet 14037ba64754436ba114928a3e0dcbe0 from peer 34a8610adef8401085b62d8fb1fbc8fe ({username='slave'} at 127.27.111.198:35931)
I20240502 01:58:07.131922 28301 tablet_copy_service.cc:161] P a962820b582e4e158a2c765c544b8e52: Beginning new tablet copy session on tablet 14037ba64754436ba114928a3e0dcbe0 from peer 34a8610adef8401085b62d8fb1fbc8fe at {username='slave'} at 127.27.111.198:35931: session id = 34a8610adef8401085b62d8fb1fbc8fe-14037ba64754436ba114928a3e0dcbe0
I20240502 01:58:07.133183 28301 tablet_copy_source_session.cc:205] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:07.136917 30537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14037ba64754436ba114928a3e0dcbe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:07.142794 30537 tablet_copy_client.cc:783] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:07.143096 30537 tablet_copy_client.cc:647] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:07.144968 30538 tablet_copy_client.cc:1002] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Injecting latency into file download: 1500
W20240502 01:58:07.286414 30548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:07.286698 30548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:07.289667 30548 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:58:07.379225 30115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.379994 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:07.382292 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.384491 28391 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:07.385459 28260 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:07.386181 29722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.399399 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.418612 28523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240502 01:58:07.452963 30243 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 28199
W20240502 01:58:07.461022 28607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
W20240502 01:58:07.461791 28607 consensus_peers.cc:485] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:07.501978 28345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
W20240502 01:58:07.502751 28345 consensus_peers.cc:485] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:07.579825 28476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
W20240502 01:58:07.580672 28476 consensus_peers.cc:485] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:07.634546 28607 consensus_peers.cc:485] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:07.636029 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.636251 30115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.640666 29722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.640715 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:07.653070 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.655210 28391 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:07.662106 28607 consensus_peers.cc:485] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:07.666249 28523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:07.754557 30115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.761049 28391 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:07.762374 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.769086 29722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.769335 28523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:07.783113 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:07.787966 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:07.798998 28345 consensus_peers.cc:485] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:07.828305 28412 consensus_queue.cc:237] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } }
I20240502 01:58:07.830032 28673 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:07.830775 28543 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:07.830658 30680 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:07.832018 30680 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:07.832654 30681 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dd5f234ed11745168cd67ee403ff098e (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } } }
I20240502 01:58:07.833070 28673 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dd5f234ed11745168cd67ee403ff098e (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } } }
I20240502 01:58:07.834559 28543 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dd5f234ed11745168cd67ee403ff098e (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } } }
W20240502 01:58:07.835808 28348 consensus_peers.cc:485] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer dd5f234ed11745168cd67ee403ff098e (127.27.111.199:42203): Couldn't send request to peer dd5f234ed11745168cd67ee403ff098e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e87ab8d205e43098dd84ae5f4eeeb6c. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:07.879266 28607 consensus_peers.cc:485] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:07.892534 28345 consensus_peers.cc:485] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:58:07.902536 28476 consensus_peers.cc:485] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:07.930029 28130 catalog_manager.cc:5484] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 reported cstate change: config changed from index -1 to 4, NON_VOTER dd5f234ed11745168cd67ee403ff098e (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } } }
I20240502 01:58:07.940256 28543 consensus_queue.cc:237] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:07.941758 28673 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:07.941967 28412 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:07.942198 29569 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:07.942427 30698 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:07.944186 29569 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } } }
I20240502 01:58:07.944617 28673 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } } }
W20240502 01:58:07.946496 28476 consensus_peers.cc:485] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 -> Peer 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198:43065): Couldn't send request to peer 34a8610adef8401085b62d8fb1fbc8fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43d194188c6f455aab0099df8ed0cbbc. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:07.946463 28412 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } } }
W20240502 01:58:07.979197 28345 consensus_peers.cc:485] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:08.033497 28134 catalog_manager.cc:5484] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 reported cstate change: config changed from index -1 to 4, NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "ef3bc5c942d04bd29540a80415308353" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } } }
I20240502 01:58:08.216722 30711 ts_tablet_manager.cc:902] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e: Initiating tablet copy from peer 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369)
I20240502 01:58:08.217476 30711 tablet_copy_client.cc:308] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:35369
I20240502 01:58:08.221168 28432 tablet_copy_service.cc:140] P 8f4edf52bb2d4624ae596b032e50c2a4: Received BeginTabletCopySession request for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c from peer dd5f234ed11745168cd67ee403ff098e ({username='slave'} at 127.27.111.199:33697)
I20240502 01:58:08.221304 28432 tablet_copy_service.cc:161] P 8f4edf52bb2d4624ae596b032e50c2a4: Beginning new tablet copy session on tablet 5e87ab8d205e43098dd84ae5f4eeeb6c from peer dd5f234ed11745168cd67ee403ff098e at {username='slave'} at 127.27.111.199:33697: session id = dd5f234ed11745168cd67ee403ff098e-5e87ab8d205e43098dd84ae5f4eeeb6c
I20240502 01:58:08.222718 28432 tablet_copy_source_session.cc:205] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:08.226781 30711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:08.232884 30711 tablet_copy_client.cc:783] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:08.233112 30711 tablet_copy_client.cc:647] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:08.234784 30715 tablet_copy_client.cc:1002] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e: tablet copy: Injecting latency into file download: 1500
I20240502 01:58:08.454047 30685 raft_consensus.cc:482] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a962820b582e4e158a2c765c544b8e52)
I20240502 01:58:08.454221 30685 raft_consensus.cc:504] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:08.454710 30685 leader_election.cc:288] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695), a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:58:08.455092 28543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5e784833984a5a95c1340c96d15994" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ef3bc5c942d04bd29540a80415308353" is_pre_election: true
W20240502 01:58:08.456291 28607 leader_election.cc:334] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
I20240502 01:58:08.456357 28607 leader_election.cc:302] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2; no voters: a962820b582e4e158a2c765c544b8e52, ef3bc5c942d04bd29540a80415308353
I20240502 01:58:08.456481 30685 raft_consensus.cc:2726] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:58:08.567065 30737 ts_tablet_manager.cc:902] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe: Initiating tablet copy from peer ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695)
I20240502 01:58:08.567567 30737 tablet_copy_client.cc:308] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:46695
I20240502 01:58:08.571302 28563 tablet_copy_service.cc:140] P ef3bc5c942d04bd29540a80415308353: Received BeginTabletCopySession request for tablet 43d194188c6f455aab0099df8ed0cbbc from peer 34a8610adef8401085b62d8fb1fbc8fe ({username='slave'} at 127.27.111.198:46493)
I20240502 01:58:08.571446 28563 tablet_copy_service.cc:161] P ef3bc5c942d04bd29540a80415308353: Beginning new tablet copy session on tablet 43d194188c6f455aab0099df8ed0cbbc from peer 34a8610adef8401085b62d8fb1fbc8fe at {username='slave'} at 127.27.111.198:46493: session id = 34a8610adef8401085b62d8fb1fbc8fe-43d194188c6f455aab0099df8ed0cbbc
I20240502 01:58:08.572698 28563 tablet_copy_source_session.cc:205] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:08.573843 30737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43d194188c6f455aab0099df8ed0cbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:08.575762 30737 tablet_copy_client.cc:783] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:08.575939 30737 tablet_copy_client.cc:647] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:08.577554 30741 tablet_copy_client.cc:1002] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Injecting latency into file download: 1500
I20240502 01:58:08.645557 30537 tablet_copy_client.cc:523] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:08.697412 30698 raft_consensus.cc:482] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a962820b582e4e158a2c765c544b8e52)
I20240502 01:58:08.697571 30698 raft_consensus.cc:504] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:08.697966 30698 leader_election.cc:288] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797), a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:58:08.698361 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5e784833984a5a95c1340c96d15994" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2" is_pre_election: true
I20240502 01:58:08.698527 28673 raft_consensus.cc:2454] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 1.
I20240502 01:58:08.698840 28479 leader_election.cc:302] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, ef3bc5c942d04bd29540a80415308353; no voters:
W20240502 01:58:08.699033 28476 leader_election.cc:334] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
I20240502 01:58:08.699029 30698 raft_consensus.cc:2781] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:58:08.699134 30698 raft_consensus.cc:482] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a962820b582e4e158a2c765c544b8e52)
I20240502 01:58:08.699191 30698 raft_consensus.cc:3037] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:08.743391 30698 raft_consensus.cc:504] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:08.743963 30698 leader_election.cc:288] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 2 election: Requested vote from peers 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797), a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:58:08.744385 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb5e784833984a5a95c1340c96d15994" candidate_uuid: "ef3bc5c942d04bd29540a80415308353" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2"
I20240502 01:58:08.744505 28673 raft_consensus.cc:3037] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:08.744581 30537 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe: Bootstrap starting.
W20240502 01:58:08.745211 28476 leader_election.cc:334] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
I20240502 01:58:08.745572 28673 raft_consensus.cc:2454] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef3bc5c942d04bd29540a80415308353 in term 2.
I20240502 01:58:08.745887 28479 leader_election.cc:302] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, ef3bc5c942d04bd29540a80415308353; no voters: a962820b582e4e158a2c765c544b8e52
I20240502 01:58:08.746121 30698 raft_consensus.cc:2781] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:08.746222 30698 raft_consensus.cc:686] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [term 2 LEADER]: Becoming Leader. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Running, Role: LEADER
I20240502 01:58:08.746357 30698 consensus_queue.cc:237] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:08.747938 30685 raft_consensus.cc:482] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a962820b582e4e158a2c765c544b8e52)
I20240502 01:58:08.748032 30685 raft_consensus.cc:504] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:08.748205 28129 catalog_manager.cc:5484] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 reported cstate change: term changed from 1 to 2, leader changed from a962820b582e4e158a2c765c544b8e52 (127.27.111.193) to ef3bc5c942d04bd29540a80415308353 (127.27.111.195). New cstate: current_term: 2 leader_uuid: "ef3bc5c942d04bd29540a80415308353" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:08.748390 30685 leader_election.cc:288] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973), 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369)
I20240502 01:58:08.748958 28412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14037ba64754436ba114928a3e0dcbe0" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" is_pre_election: true
W20240502 01:58:08.749527 28607 leader_election.cc:334] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
I20240502 01:58:08.749599 28607 leader_election.cc:302] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2; no voters: 8f4edf52bb2d4624ae596b032e50c2a4, a962820b582e4e158a2c765c544b8e52
I20240502 01:58:08.749722 30685 raft_consensus.cc:2726] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:58:08.751633 30537 log.cc:824] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:08.759320 30680 raft_consensus.cc:482] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a962820b582e4e158a2c765c544b8e52)
I20240502 01:58:08.759430 30680 raft_consensus.cc:504] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:08.759819 30680 leader_election.cc:288] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973), 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:08.760325 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14037ba64754436ba114928a3e0dcbe0" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2" is_pre_election: true
I20240502 01:58:08.760455 28673 raft_consensus.cc:2454] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 1.
I20240502 01:58:08.760748 28348 leader_election.cc:302] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, 8f4edf52bb2d4624ae596b032e50c2a4; no voters:
W20240502 01:58:08.760912 28345 leader_election.cc:334] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
I20240502 01:58:08.760982 30680 raft_consensus.cc:2781] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:58:08.761037 30680 raft_consensus.cc:482] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a962820b582e4e158a2c765c544b8e52)
I20240502 01:58:08.761080 30680 raft_consensus.cc:3037] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:08.761895 30680 raft_consensus.cc:504] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:08.762205 30680 leader_election.cc:288] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 2 election: Requested vote from peers a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973), 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:08.762702 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14037ba64754436ba114928a3e0dcbe0" candidate_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8cb8626af432419c922ceefff85f69f2"
I20240502 01:58:08.762810 28673 raft_consensus.cc:3037] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:58:08.763099 28345 leader_election.cc:334] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
I20240502 01:58:08.763664 28673 raft_consensus.cc:2454] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f4edf52bb2d4624ae596b032e50c2a4 in term 2.
I20240502 01:58:08.763919 28348 leader_election.cc:302] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, 8f4edf52bb2d4624ae596b032e50c2a4; no voters: a962820b582e4e158a2c765c544b8e52
I20240502 01:58:08.764132 30680 raft_consensus.cc:2781] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:08.764217 30680 raft_consensus.cc:686] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 LEADER]: Becoming Leader. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Running, Role: LEADER
I20240502 01:58:08.764355 30680 consensus_queue.cc:237] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:08.766425 28129 catalog_manager.cc:5484] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 reported cstate change: term changed from 1 to 2, leader changed from a962820b582e4e158a2c765c544b8e52 (127.27.111.193) to 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194). New cstate: current_term: 2 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:08.777279 30537 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:08.777577 30537 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe: Bootstrap complete.
I20240502 01:58:08.777859 30537 ts_tablet_manager.cc:1367] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe: Time spent bootstrapping tablet: real 0.033s user 0.029s sys 0.000s
I20240502 01:58:08.780314 30537 raft_consensus.cc:348] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:08.780532 30537 raft_consensus.cc:729] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 34a8610adef8401085b62d8fb1fbc8fe, State: Initialized, Role: LEARNER
I20240502 01:58:08.780841 30537 consensus_queue.cc:260] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:08.781603 30537 ts_tablet_manager.cc:1398] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20240502 01:58:08.782434 30537 tablet_copy_client.cc:1107] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
W20240502 01:58:09.177470 28476 consensus_peers.cc:485] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:09.276113 28673 raft_consensus.cc:1261] T cb5e784833984a5a95c1340c96d15994 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:58:09.276574 30774 consensus_queue.cc:1035] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:09.311939 28673 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:09.312368 30680 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:58:09.317112 28345 consensus_peers.cc:485] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:09.319712 29873 raft_consensus.cc:3037] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 1 LEARNER]: Advancing to term 2
I20240502 01:58:09.320878 29873 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 LEARNER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:09.321393 30686 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:09.324841 30686 raft_consensus.cc:1050] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: attempting to promote NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe to VOTER
I20240502 01:58:09.325296 30686 consensus_queue.cc:237] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } }
I20240502 01:58:09.326831 29873 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 LEARNER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:58:09.327201 30681 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:09.327276 28673 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:58:09.327674 30681 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:58:09.329304 28345 consensus_peers.cc:485] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:09.329663 30801 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:09.330018 29873 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:09.331552 28673 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:09.332288 28130 catalog_manager.cc:5484] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe reported cstate change: config changed from index 5 to 7, 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:09.336088 28412 consensus_queue.cc:237] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } }
I20240502 01:58:09.337152 28673 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:58:09.337297 29873 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:58:09.337499 30686 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:09.337625 30681 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:09.339529 28673 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER a962820b582e4e158a2c765c544b8e52 (127.27.111.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:09.339129 30686 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER a962820b582e4e158a2c765c544b8e52 (127.27.111.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:09.340023 29873 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER a962820b582e4e158a2c765c544b8e52 (127.27.111.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:09.341674 28122 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 14037ba64754436ba114928a3e0dcbe0 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:09.341835 28134 catalog_manager.cc:5484] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 reported cstate change: config changed from index 7 to 8, VOTER a962820b582e4e158a2c765c544b8e52 (127.27.111.193) evicted. New cstate: current_term: 2 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
W20240502 01:58:09.344630 28122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
W20240502 01:58:09.345513 28122 catalog_manager.cc:4573] TS a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 14037ba64754436ba114928a3e0dcbe0: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
W20240502 01:58:09.731254 28607 consensus_peers.cc:485] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:09.735246 30711 tablet_copy_client.cc:523] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:09.737196 30711 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e: Bootstrap starting.
I20240502 01:58:09.742964 30711 log.cc:824] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:09.764259 30711 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:09.764631 30711 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e: Bootstrap complete.
I20240502 01:58:09.764935 30711 ts_tablet_manager.cc:1367] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e: Time spent bootstrapping tablet: real 0.028s user 0.027s sys 0.000s
I20240502 01:58:09.767072 30711 raft_consensus.cc:348] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } }
I20240502 01:58:09.767287 30711 raft_consensus.cc:729] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dd5f234ed11745168cd67ee403ff098e, State: Initialized, Role: LEARNER
I20240502 01:58:09.767630 30711 consensus_queue.cc:260] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } }
I20240502 01:58:09.768361 30711 ts_tablet_manager.cc:1398] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 01:58:09.769048 28432 tablet_copy_service.cc:342] P 8f4edf52bb2d4624ae596b032e50c2a4: Request end of tablet copy session dd5f234ed11745168cd67ee403ff098e-5e87ab8d205e43098dd84ae5f4eeeb6c received from {username='slave'} at 127.27.111.199:33697
I20240502 01:58:09.769167 28432 tablet_copy_service.cc:433] P 8f4edf52bb2d4624ae596b032e50c2a4: ending tablet copy session dd5f234ed11745168cd67ee403ff098e-5e87ab8d205e43098dd84ae5f4eeeb6c on tablet 5e87ab8d205e43098dd84ae5f4eeeb6c with peer dd5f234ed11745168cd67ee403ff098e
W20240502 01:58:10.024858 28345 consensus_peers.cc:485] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:10.077857 30737 tablet_copy_client.cc:523] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:10.079483 30737 tablet_bootstrap.cc:492] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe: Bootstrap starting.
W20240502 01:58:10.081185 28607 consensus_peers.cc:485] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:10.098723 30737 tablet_bootstrap.cc:492] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:10.099083 30737 tablet_bootstrap.cc:492] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe: Bootstrap complete.
I20240502 01:58:10.099260 30737 ts_tablet_manager.cc:1367] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
I20240502 01:58:10.099905 30737 raft_consensus.cc:348] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:10.100037 30737 raft_consensus.cc:729] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 34a8610adef8401085b62d8fb1fbc8fe, State: Initialized, Role: LEARNER
I20240502 01:58:10.100127 30737 consensus_queue.cc:260] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:10.100436 30737 ts_tablet_manager.cc:1398] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:10.101217 28563 tablet_copy_service.cc:342] P ef3bc5c942d04bd29540a80415308353: Request end of tablet copy session 34a8610adef8401085b62d8fb1fbc8fe-43d194188c6f455aab0099df8ed0cbbc received from {username='slave'} at 127.27.111.198:46493
I20240502 01:58:10.101336 28563 tablet_copy_service.cc:433] P ef3bc5c942d04bd29540a80415308353: ending tablet copy session 34a8610adef8401085b62d8fb1fbc8fe-43d194188c6f455aab0099df8ed0cbbc on tablet 43d194188c6f455aab0099df8ed0cbbc with peer 34a8610adef8401085b62d8fb1fbc8fe
W20240502 01:58:10.203397 28476 consensus_peers.cc:485] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:10.213622 28607 consensus_peers.cc:485] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:10.219535 28345 consensus_peers.cc:485] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:10.228027 28607 consensus_peers.cc:485] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:10.278807 30004 raft_consensus.cc:1203] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 01:58:10.354104 28345 consensus_peers.cc:485] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:10.433676 28476 consensus_peers.cc:485] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:10.449430 29873 raft_consensus.cc:1203] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 01:58:10.555610 28122 catalog_manager.cc:4573] TS a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 14037ba64754436ba114928a3e0dcbe0: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
W20240502 01:58:10.559923 28345 consensus_peers.cc:485] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:58:10.692797 30875 raft_consensus.cc:1050] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4: attempting to promote NON_VOTER dd5f234ed11745168cd67ee403ff098e to VOTER
I20240502 01:58:10.693235 30875 consensus_queue.cc:237] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } }
I20240502 01:58:10.694773 28673 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:10.694839 30004 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 1 LEARNER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:10.695168 28543 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:10.695334 30877 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:10.695693 30875 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:10.695938 30801 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:10.697772 30801 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:10.698247 28543 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:10.699286 30004 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:10.700455 28130 catalog_manager.cc:5484] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 reported cstate change: config changed from index 4 to 5, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:10.701975 28673 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:10.704550 28412 consensus_queue.cc:237] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } }
I20240502 01:58:10.705592 30004 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:10.705595 28673 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:10.705920 30874 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:10.706084 30875 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:10.707628 30877 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef3bc5c942d04bd29540a80415308353 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:10.708117 28673 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef3bc5c942d04bd29540a80415308353 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:10.708125 30004 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ef3bc5c942d04bd29540a80415308353 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:10.709067 28122 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:10.710533 28134 catalog_manager.cc:5484] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e reported cstate change: config changed from index 5 to 6, VOTER ef3bc5c942d04bd29540a80415308353 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:10.713037 28523 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c with delete_type TABLET_DATA_TOMBSTONED (TS ef3bc5c942d04bd29540a80415308353 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51302
I20240502 01:58:10.713752 30885 tablet_replica.cc:330] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353: stopping tablet replica
I20240502 01:58:10.714113 30885 raft_consensus.cc:2229] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:10.714424 30885 raft_consensus.cc:2258] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:10.715472 30885 ts_tablet_manager.cc:1875] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:10.717442 30885 ts_tablet_manager.cc:1888] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:10.717530 30885 log.cc:1196] T 5e87ab8d205e43098dd84ae5f4eeeb6c P ef3bc5c942d04bd29540a80415308353: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/5e87ab8d205e43098dd84ae5f4eeeb6c
I20240502 01:58:10.718078 28124 catalog_manager.cc:4830] TS ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695): tablet 5e87ab8d205e43098dd84ae5f4eeeb6c (table rebalance_test_table_1 [id=eae4e7ef163a4d2788386ab9c9fd7841]) successfully deleted
I20240502 01:58:10.822695 28543 consensus_queue.cc:237] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: true } }
I20240502 01:58:10.824676 28673 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:10.824819 28412 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:58:10.825181 30903 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:10.825392 30902 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:58:10.827363 30903 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: true } } }
I20240502 01:58:10.827765 28673 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: true } } }
W20240502 01:58:10.829123 28479 consensus_peers.cc:485] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 -> Peer a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197:40085): Couldn't send request to peer a1a0ca3ab1ca4872b2bb4395068b7033. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30179e5d45eb47cd88045b81e4872c78. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:10.830763 28412 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: true } } }
I20240502 01:58:10.830843 28130 catalog_manager.cc:5484] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 reported cstate change: config changed from index -1 to 4, NON_VOTER a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "ef3bc5c942d04bd29540a80415308353" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:10.986953 30697 raft_consensus.cc:1050] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353: attempting to promote NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe to VOTER
I20240502 01:58:10.987536 30697 consensus_queue.cc:237] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } }
I20240502 01:58:10.990273 28673 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:10.990270 28412 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:10.990789 29873 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 1 LEARNER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:10.990769 30902 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:10.991353 30903 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:10.991688 30903 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:10.993660 30903 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:10.994269 29873 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:10.995816 28412 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:10.996099 28673 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:10.998212 28129 catalog_manager.cc:5484] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe reported cstate change: config changed from index 4 to 5, 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef3bc5c942d04bd29540a80415308353" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:11.002249 28543 consensus_queue.cc:237] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } }
I20240502 01:58:11.004094 28673 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:11.004573 29873 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:58:11.004971 30903 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:11.005645 30903 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:11.006618 30902 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:11.007091 28673 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:11.008082 28124 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 43d194188c6f455aab0099df8ed0cbbc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:11.008863 29873 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:11.009711 28132 catalog_manager.cc:5484] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 reported cstate change: config changed from index 5 to 6, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "ef3bc5c942d04bd29540a80415308353" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:11.012756 28391 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d194188c6f455aab0099df8ed0cbbc with delete_type TABLET_DATA_TOMBSTONED (TS 8f4edf52bb2d4624ae596b032e50c2a4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36436
I20240502 01:58:11.013540 30924 tablet_replica.cc:330] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4: stopping tablet replica
I20240502 01:58:11.013926 30924 raft_consensus.cc:2229] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:11.014179 30924 raft_consensus.cc:2258] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:11.015527 30924 ts_tablet_manager.cc:1875] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:11.018215 30924 ts_tablet_manager.cc:1888] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:58:11.018326 30924 log.cc:1196] T 43d194188c6f455aab0099df8ed0cbbc P 8f4edf52bb2d4624ae596b032e50c2a4: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/43d194188c6f455aab0099df8ed0cbbc
I20240502 01:58:11.019014 28122 catalog_manager.cc:4830] TS 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369): tablet 43d194188c6f455aab0099df8ed0cbbc (table rebalance_test_table_1 [id=eae4e7ef163a4d2788386ab9c9fd7841]) successfully deleted
I20240502 01:58:11.352675 30945 ts_tablet_manager.cc:902] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033: Initiating tablet copy from peer ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695)
I20240502 01:58:11.353462 30945 tablet_copy_client.cc:308] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:46695
I20240502 01:58:11.357594 28563 tablet_copy_service.cc:140] P ef3bc5c942d04bd29540a80415308353: Received BeginTabletCopySession request for tablet 30179e5d45eb47cd88045b81e4872c78 from peer a1a0ca3ab1ca4872b2bb4395068b7033 ({username='slave'} at 127.27.111.197:42297)
I20240502 01:58:11.357720 28563 tablet_copy_service.cc:161] P ef3bc5c942d04bd29540a80415308353: Beginning new tablet copy session on tablet 30179e5d45eb47cd88045b81e4872c78 from peer a1a0ca3ab1ca4872b2bb4395068b7033 at {username='slave'} at 127.27.111.197:42297: session id = a1a0ca3ab1ca4872b2bb4395068b7033-30179e5d45eb47cd88045b81e4872c78
I20240502 01:58:11.358566 28563 tablet_copy_source_session.cc:205] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:11.361846 30945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30179e5d45eb47cd88045b81e4872c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:11.367671 30945 tablet_copy_client.cc:783] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:11.367863 30945 tablet_copy_client.cc:647] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:11.369381 30949 tablet_copy_client.cc:1002] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033: tablet copy: Injecting latency into file download: 1500
W20240502 01:58:11.515713 28476 consensus_peers.cc:485] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:58:11.623939 28122 catalog_manager.cc:4573] TS a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 14037ba64754436ba114928a3e0dcbe0: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
W20240502 01:58:12.113710 28607 consensus_peers.cc:485] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:12.527846 28607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 01:58:12.543074 28345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 01:58:12.591553 28345 consensus_peers.cc:485] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:12.637918 28476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 01:58:12.669293 28345 consensus_peers.cc:485] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:12.702989 28607 consensus_peers.cc:485] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:12.744161 28345 consensus_peers.cc:485] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:12.763423 28607 consensus_peers.cc:485] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:12.774565 28607 consensus_peers.cc:485] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:12.846493 28476 consensus_peers.cc:485] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:12.859179 28476 consensus_peers.cc:485] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:12.869875 30945 tablet_copy_client.cc:523] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:12.872047 30945 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033: Bootstrap starting.
I20240502 01:58:12.877269 30945 log.cc:824] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:12.897034 30945 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:12.897445 30945 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033: Bootstrap complete.
I20240502 01:58:12.897738 30945 ts_tablet_manager.cc:1367] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033: Time spent bootstrapping tablet: real 0.026s user 0.027s sys 0.000s
I20240502 01:58:12.899842 30945 raft_consensus.cc:348] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: true } }
I20240502 01:58:12.900074 30945 raft_consensus.cc:729] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a1a0ca3ab1ca4872b2bb4395068b7033, State: Initialized, Role: LEARNER
I20240502 01:58:12.900425 30945 consensus_queue.cc:260] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: true } }
I20240502 01:58:12.901269 30945 ts_tablet_manager.cc:1398] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:58:12.901875 28563 tablet_copy_service.cc:342] P ef3bc5c942d04bd29540a80415308353: Request end of tablet copy session a1a0ca3ab1ca4872b2bb4395068b7033-30179e5d45eb47cd88045b81e4872c78 received from {username='slave'} at 127.27.111.197:42297
I20240502 01:58:12.901950 28563 tablet_copy_service.cc:433] P ef3bc5c942d04bd29540a80415308353: ending tablet copy session a1a0ca3ab1ca4872b2bb4395068b7033-30179e5d45eb47cd88045b81e4872c78 on tablet 30179e5d45eb47cd88045b81e4872c78 with peer a1a0ca3ab1ca4872b2bb4395068b7033
I20240502 01:58:12.904160 29742 raft_consensus.cc:1203] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 01:58:13.045892 28345 consensus_peers.cc:485] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:58:13.334148 31006 raft_consensus.cc:1050] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353: attempting to promote NON_VOTER a1a0ca3ab1ca4872b2bb4395068b7033 to VOTER
I20240502 01:58:13.334604 31006 consensus_queue.cc:237] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } }
I20240502 01:58:13.336122 28673 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:13.336217 29742 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 1 LEARNER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:13.336455 28412 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:58:13.336757 31005 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:13.336913 31008 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:13.337144 30903 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:58:13.339777 31005 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:13.340349 29742 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:13.341892 28412 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:13.342216 28673 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:13.342787 28130 catalog_manager.cc:5484] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 reported cstate change: config changed from index 4 to 5, a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef3bc5c942d04bd29540a80415308353" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:13.350849 28543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78"
dest_uuid: "ef3bc5c942d04bd29540a80415308353"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56878
I20240502 01:58:13.351016 28543 raft_consensus.cc:595] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.563050 28543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78"
dest_uuid: "ef3bc5c942d04bd29540a80415308353"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56884
I20240502 01:58:13.563221 28543 raft_consensus.cc:595] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Received request to transfer leadership
W20240502 01:58:13.722776 28122 catalog_manager.cc:4573] TS a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 14037ba64754436ba114928a3e0dcbe0: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
I20240502 01:58:13.774381 28543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78"
dest_uuid: "ef3bc5c942d04bd29540a80415308353"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56906
I20240502 01:58:13.774556 28543 raft_consensus.cc:595] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:13.787199 31006 raft_consensus.cc:979] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353: : Instructing follower 8cb8626af432419c922ceefff85f69f2 to start an election
I20240502 01:58:13.787307 31006 raft_consensus.cc:1067] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Signalling peer 8cb8626af432419c922ceefff85f69f2 to start an election
I20240502 01:58:13.787823 28673 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78"
dest_uuid: "8cb8626af432419c922ceefff85f69f2"
from {username='slave'} at 127.27.111.195:43819
I20240502 01:58:13.787982 28673 raft_consensus.cc:482] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:13.788031 28673 raft_consensus.cc:3037] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:13.789024 28673 raft_consensus.cc:504] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } }
I20240502 01:58:13.789453 28673 leader_election.cc:288] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 2 election: Requested vote from peers 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369), ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695), a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197:40085)
I20240502 01:58:13.789732 28543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ef3bc5c942d04bd29540a80415308353"
I20240502 01:58:13.789846 28543 raft_consensus.cc:3032] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:13.789812 28412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
I20240502 01:58:13.789881 28543 raft_consensus.cc:729] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Running, Role: LEADER
I20240502 01:58:13.789956 28412 raft_consensus.cc:3037] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:13.790046 28543 consensus_queue.cc:260] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } }
I20240502 01:58:13.790277 28543 raft_consensus.cc:3037] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:13.791023 28412 raft_consensus.cc:2454] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 2.
I20240502 01:58:13.791364 28543 raft_consensus.cc:2454] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 2.
I20240502 01:58:13.791656 28609 leader_election.cc:302] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, 8f4edf52bb2d4624ae596b032e50c2a4, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:13.791841 30986 raft_consensus.cc:2781] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:13.792013 30986 raft_consensus.cc:686] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Becoming Leader. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Running, Role: LEADER
I20240502 01:58:13.792162 30986 consensus_queue.cc:237] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } }
I20240502 01:58:13.794176 29742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033"
I20240502 01:58:13.794342 29742 raft_consensus.cc:3037] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:13.794232 28134 catalog_manager.cc:5484] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 reported cstate change: term changed from 1 to 2, leader changed from ef3bc5c942d04bd29540a80415308353 (127.27.111.195) to 8cb8626af432419c922ceefff85f69f2 (127.27.111.196). New cstate: current_term: 2 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:13.795307 29742 raft_consensus.cc:2454] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 2.
I20240502 01:58:14.315913 28543 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:14.316407 31064 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:14.322160 29742 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:14.322576 31064 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:58:14.326072 28412 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:58:14.326831 31067 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:58:14.362946 28476 consensus_peers.cc:485] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:58:14.484057 28607 consensus_peers.cc:485] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:14.864887 28673 consensus_queue.cc:237] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } }
W20240502 01:58:14.866046 28345 consensus_peers.cc:485] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:14.866278 29742 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:58:14.866317 28412 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:58:14.866724 31087 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:14.866936 31067 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:14.868464 31088 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ef3bc5c942d04bd29540a80415308353 (127.27.111.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:14.868782 29742 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ef3bc5c942d04bd29540a80415308353 (127.27.111.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:14.868906 28412 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER ef3bc5c942d04bd29540a80415308353 (127.27.111.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:14.870168 28125 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 30179e5d45eb47cd88045b81e4872c78 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:14.871333 28134 catalog_manager.cc:5484] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 reported cstate change: config changed from index 5 to 7, VOTER ef3bc5c942d04bd29540a80415308353 (127.27.111.195) evicted. New cstate: current_term: 2 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:14.874362 28523 tablet_service.cc:1514] Processing DeleteTablet for tablet 30179e5d45eb47cd88045b81e4872c78 with delete_type TABLET_DATA_TOMBSTONED (TS ef3bc5c942d04bd29540a80415308353 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51302
I20240502 01:58:14.874753 31094 tablet_replica.cc:330] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353: stopping tablet replica
I20240502 01:58:14.874876 31094 raft_consensus.cc:2229] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:14.874994 31094 raft_consensus.cc:2258] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:14.875591 31094 ts_tablet_manager.cc:1875] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:14.877449 31094 ts_tablet_manager.cc:1888] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:58:14.877521 31094 log.cc:1196] T 30179e5d45eb47cd88045b81e4872c78 P ef3bc5c942d04bd29540a80415308353: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/30179e5d45eb47cd88045b81e4872c78
I20240502 01:58:14.878028 28124 catalog_manager.cc:4830] TS ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695): tablet 30179e5d45eb47cd88045b81e4872c78 (table rebalance_test_table_3 [id=86cf304e1bdb4ceabae33570f6c0d7fc]) successfully deleted
W20240502 01:58:14.988934 28345 consensus_peers.cc:485] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:15.053611 28607 consensus_peers.cc:485] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:15.070869 30115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:15.072024 28654 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:15.074563 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:15.076113 29722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:15.083287 28391 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:15.083356 28523 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:15.084203 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:15.105641 28607 consensus_peers.cc:485] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:15.130894 28673 consensus_queue.cc:237] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } }
I20240502 01:58:15.132670 28412 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:58:15.133092 29742 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:58:15.133136 31088 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:15.133448 31067 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:15.135166 31087 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER dd5f234ed11745168cd67ee403ff098e (127.27.111.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } } }
I20240502 01:58:15.135638 29742 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER dd5f234ed11745168cd67ee403ff098e (127.27.111.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } } }
W20240502 01:58:15.137539 28610 consensus_peers.cc:485] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 -> Peer dd5f234ed11745168cd67ee403ff098e (127.27.111.199:42203): Couldn't send request to peer dd5f234ed11745168cd67ee403ff098e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30179e5d45eb47cd88045b81e4872c78. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:15.138065 28130 catalog_manager.cc:5484] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 reported cstate change: config changed from index 7 to 8, NON_VOTER dd5f234ed11745168cd67ee403ff098e (127.27.111.199) added. New cstate: current_term: 2 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:15.138887 28412 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER dd5f234ed11745168cd67ee403ff098e (127.27.111.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } } }
I20240502 01:58:15.147182 28543 consensus_queue.cc:237] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } }
I20240502 01:58:15.148783 28673 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:15.148785 29873 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:15.149199 31006 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:15.149435 31150 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:15.152201 31006 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } } }
W20240502 01:58:15.152536 28478 consensus_peers.cc:485] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 -> Peer 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200:39123): Couldn't send request to peer 0773aa6e61f442bb86d966ae9e97cbc6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43d194188c6f455aab0099df8ed0cbbc. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:15.152746 29873 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } } }
I20240502 01:58:15.152947 28673 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } } }
I20240502 01:58:15.154800 28130 catalog_manager.cc:5484] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 reported cstate change: config changed from index 6 to 7, NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "ef3bc5c942d04bd29540a80415308353" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:15.246371 28607 consensus_peers.cc:485] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:15.408267 28345 consensus_peers.cc:485] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:15.425964 28345 consensus_peers.cc:485] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:15.468806 28476 consensus_peers.cc:485] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:15.502671 28476 consensus_peers.cc:485] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:58:15.625108 31185 ts_tablet_manager.cc:902] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6: Initiating tablet copy from peer ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695)
I20240502 01:58:15.625821 31185 tablet_copy_client.cc:308] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:46695
I20240502 01:58:15.629249 28563 tablet_copy_service.cc:140] P ef3bc5c942d04bd29540a80415308353: Received BeginTabletCopySession request for tablet 43d194188c6f455aab0099df8ed0cbbc from peer 0773aa6e61f442bb86d966ae9e97cbc6 ({username='slave'} at 127.27.111.200:49829)
I20240502 01:58:15.629343 28563 tablet_copy_service.cc:161] P ef3bc5c942d04bd29540a80415308353: Beginning new tablet copy session on tablet 43d194188c6f455aab0099df8ed0cbbc from peer 0773aa6e61f442bb86d966ae9e97cbc6 at {username='slave'} at 127.27.111.200:49829: session id = 0773aa6e61f442bb86d966ae9e97cbc6-43d194188c6f455aab0099df8ed0cbbc
I20240502 01:58:15.630095 28563 tablet_copy_source_session.cc:205] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:15.633282 31185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43d194188c6f455aab0099df8ed0cbbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:15.638332 31185 tablet_copy_client.cc:783] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:15.638535 31185 tablet_copy_client.cc:647] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:15.639720 31189 tablet_copy_client.cc:1002] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Injecting latency into file download: 1500
I20240502 01:58:15.752056 31191 ts_tablet_manager.cc:902] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e: Initiating tablet copy from peer 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:15.752527 31191 tablet_copy_client.cc:308] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:40797
I20240502 01:58:15.755892 28694 tablet_copy_service.cc:140] P 8cb8626af432419c922ceefff85f69f2: Received BeginTabletCopySession request for tablet 30179e5d45eb47cd88045b81e4872c78 from peer dd5f234ed11745168cd67ee403ff098e ({username='slave'} at 127.27.111.199:52927)
I20240502 01:58:15.756018 28694 tablet_copy_service.cc:161] P 8cb8626af432419c922ceefff85f69f2: Beginning new tablet copy session on tablet 30179e5d45eb47cd88045b81e4872c78 from peer dd5f234ed11745168cd67ee403ff098e at {username='slave'} at 127.27.111.199:52927: session id = dd5f234ed11745168cd67ee403ff098e-30179e5d45eb47cd88045b81e4872c78
I20240502 01:58:15.757294 28694 tablet_copy_source_session.cc:205] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:15.758484 31191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30179e5d45eb47cd88045b81e4872c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:15.760239 31191 tablet_copy_client.cc:783] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:15.760401 31191 tablet_copy_client.cc:647] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:15.761732 31194 tablet_copy_client.cc:1002] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e: tablet copy: Injecting latency into file download: 1500
W20240502 01:58:16.936975 28476 consensus_peers.cc:485] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:58:16.937623 28607 consensus_peers.cc:485] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:58:17.140173 31185 tablet_copy_client.cc:523] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:17.142292 31185 tablet_bootstrap.cc:492] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6: Bootstrap starting.
I20240502 01:58:17.148365 31185 log.cc:824] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:17.168787 31185 tablet_bootstrap.cc:492] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:17.169164 31185 tablet_bootstrap.cc:492] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6: Bootstrap complete.
I20240502 01:58:17.169485 31185 ts_tablet_manager.cc:1367] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6: Time spent bootstrapping tablet: real 0.027s user 0.026s sys 0.000s
I20240502 01:58:17.171525 31185 raft_consensus.cc:348] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } }
I20240502 01:58:17.171736 31185 raft_consensus.cc:729] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0773aa6e61f442bb86d966ae9e97cbc6, State: Initialized, Role: LEARNER
I20240502 01:58:17.172060 31185 consensus_queue.cc:260] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } }
I20240502 01:58:17.172835 31185 ts_tablet_manager.cc:1398] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240502 01:58:17.173564 28563 tablet_copy_service.cc:342] P ef3bc5c942d04bd29540a80415308353: Request end of tablet copy session 0773aa6e61f442bb86d966ae9e97cbc6-43d194188c6f455aab0099df8ed0cbbc received from {username='slave'} at 127.27.111.200:49829
I20240502 01:58:17.173704 28563 tablet_copy_service.cc:433] P ef3bc5c942d04bd29540a80415308353: ending tablet copy session 0773aa6e61f442bb86d966ae9e97cbc6-43d194188c6f455aab0099df8ed0cbbc on tablet 43d194188c6f455aab0099df8ed0cbbc with peer 0773aa6e61f442bb86d966ae9e97cbc6
I20240502 01:58:17.198704 30135 raft_consensus.cc:1203] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:58:17.262058 31191 tablet_copy_client.cc:523] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:17.263543 31191 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e: Bootstrap starting.
I20240502 01:58:17.283316 31191 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:17.283629 31191 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e: Bootstrap complete.
I20240502 01:58:17.283804 31191 ts_tablet_manager.cc:1367] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.003s
I20240502 01:58:17.284338 31191 raft_consensus.cc:348] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } }
I20240502 01:58:17.284466 31191 raft_consensus.cc:729] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: dd5f234ed11745168cd67ee403ff098e, State: Initialized, Role: LEARNER
I20240502 01:58:17.284566 31191 consensus_queue.cc:260] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } }
I20240502 01:58:17.284930 31191 ts_tablet_manager.cc:1398] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:17.285598 28694 tablet_copy_service.cc:342] P 8cb8626af432419c922ceefff85f69f2: Request end of tablet copy session dd5f234ed11745168cd67ee403ff098e-30179e5d45eb47cd88045b81e4872c78 received from {username='slave'} at 127.27.111.199:52927
I20240502 01:58:17.285719 28694 tablet_copy_service.cc:433] P 8cb8626af432419c922ceefff85f69f2: ending tablet copy session dd5f234ed11745168cd67ee403ff098e-30179e5d45eb47cd88045b81e4872c78 on tablet 30179e5d45eb47cd88045b81e4872c78 with peer dd5f234ed11745168cd67ee403ff098e
W20240502 01:58:17.354157 28607 consensus_peers.cc:485] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:58:17.356979 30004 raft_consensus.cc:1203] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20240502 01:58:17.470948 28607 consensus_peers.cc:485] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:58:17.507691 28345 consensus_peers.cc:485] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:58:17.546851 28345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 01:58:17.547621 28345 consensus_peers.cc:485] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:58:17.746320 28607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 01:58:17.746982 28607 consensus_peers.cc:485] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:58:17.762672 31305 raft_consensus.cc:1050] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353: attempting to promote NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 to VOTER
I20240502 01:58:17.763095 31305 consensus_queue.cc:237] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:17.764557 28674 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:17.764576 29873 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 1 FOLLOWER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:17.764763 30135 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [term 1 LEARNER]: Refusing update from remote peer ef3bc5c942d04bd29540a80415308353: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:17.764948 31308 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:17.765292 31217 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:17.766067 31305 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:17.768173 31305 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:17.768708 30135 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:17.770340 28674 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:17.770972 29873 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:17.771384 28132 catalog_manager.cc:5484] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 reported cstate change: config changed from index 7 to 8, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef3bc5c942d04bd29540a80415308353" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
W20240502 01:58:17.841185 28122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 01:58:17.842012 28122 catalog_manager.cc:4573] TS a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 14037ba64754436ba114928a3e0dcbe0: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
W20240502 01:58:17.874502 28476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 01:58:17.875205 28476 consensus_peers.cc:485] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:58:17.914031 31321 raft_consensus.cc:1050] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2: attempting to promote NON_VOTER dd5f234ed11745168cd67ee403ff098e to VOTER
I20240502 01:58:17.914554 31321 consensus_queue.cc:237] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } }
I20240502 01:58:17.916040 30004 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 2 LEARNER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:17.916121 29742 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:17.916117 28412 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:17.916517 31230 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:17.916749 31323 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:17.916909 31321 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:17.919057 31230 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:17.919974 29742 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:17.920128 30004 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:17.922195 28412 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:17.923432 28134 catalog_manager.cc:5484] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 reported cstate change: config changed from index 8 to 9, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:17.926553 28674 consensus_queue.cc:237] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } }
I20240502 01:58:17.927968 30004 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:17.928231 29742 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:17.928334 31320 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:17.929641 31323 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:17.930848 31321 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:17.931205 30004 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:17.932149 29742 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:17.933583 28125 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 30179e5d45eb47cd88045b81e4872c78 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:17.933761 28132 catalog_manager.cc:5484] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e reported cstate change: config changed from index 9 to 10, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New cstate: current_term: 2 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:17.936697 28391 tablet_service.cc:1514] Processing DeleteTablet for tablet 30179e5d45eb47cd88045b81e4872c78 with delete_type TABLET_DATA_TOMBSTONED (TS 8f4edf52bb2d4624ae596b032e50c2a4 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36436
I20240502 01:58:17.937085 31339 tablet_replica.cc:330] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4: stopping tablet replica
I20240502 01:58:17.937319 31339 raft_consensus.cc:2229] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:17.937635 31339 raft_consensus.cc:2258] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:17.937602 28543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "43d194188c6f455aab0099df8ed0cbbc"
dest_uuid: "ef3bc5c942d04bd29540a80415308353"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57046
I20240502 01:58:17.937729 28543 raft_consensus.cc:595] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:17.938817 31339 ts_tablet_manager.cc:1875] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:17.941071 31339 ts_tablet_manager.cc:1888] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:58:17.941195 31339 log.cc:1196] T 30179e5d45eb47cd88045b81e4872c78 P 8f4edf52bb2d4624ae596b032e50c2a4: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/30179e5d45eb47cd88045b81e4872c78
I20240502 01:58:17.941745 28122 catalog_manager.cc:4830] TS 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369): tablet 30179e5d45eb47cd88045b81e4872c78 (table rebalance_test_table_3 [id=86cf304e1bdb4ceabae33570f6c0d7fc]) successfully deleted
I20240502 01:58:17.951907 28412 consensus_queue.cc:237] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } }
I20240502 01:58:17.953366 28674 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240502 01:58:17.953547 28348 consensus_peers.cc:485] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer dd5f234ed11745168cd67ee403ff098e (127.27.111.199:42203): Couldn't send request to peer dd5f234ed11745168cd67ee403ff098e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14037ba64754436ba114928a3e0dcbe0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:17.953577 29873 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:17.953920 31353 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:17.954151 31352 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:17.955942 30801 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER dd5f234ed11745168cd67ee403ff098e (127.27.111.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } } }
I20240502 01:58:17.956375 29873 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER dd5f234ed11745168cd67ee403ff098e (127.27.111.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } } }
I20240502 01:58:17.958524 28136 catalog_manager.cc:5484] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 reported cstate change: config changed from index 8 to 9, NON_VOTER dd5f234ed11745168cd67ee403ff098e (127.27.111.199) added. New cstate: current_term: 2 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:17.959504 28674 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER dd5f234ed11745168cd67ee403ff098e (127.27.111.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } } }
I20240502 01:58:17.967691 28412 consensus_queue.cc:237] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: true } }
I20240502 01:58:17.969100 30004 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:17.969508 28674 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:58:17.969594 31353 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:17.969894 30801 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:58:17.971666 31353 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: true } } }
I20240502 01:58:17.972033 30004 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: true } } }
I20240502 01:58:17.974020 28674 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: true } } }
I20240502 01:58:17.974347 28136 catalog_manager.cc:5484] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 reported cstate change: config changed from index 6 to 7, NON_VOTER a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:17.976198 28348 consensus_peers.cc:485] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197:40085): Couldn't send request to peer a1a0ca3ab1ca4872b2bb4395068b7033. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e87ab8d205e43098dd84ae5f4eeeb6c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:17.991739 28543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "43d194188c6f455aab0099df8ed0cbbc"
dest_uuid: "ef3bc5c942d04bd29540a80415308353"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57072
I20240502 01:58:17.991883 28543 raft_consensus.cc:595] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Received request to transfer leadership
W20240502 01:58:18.000752 28345 consensus_peers.cc:485] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:58:18.060325 28476 consensus_peers.cc:485] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:58:18.188258 31307 raft_consensus.cc:979] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353: : Instructing follower 0773aa6e61f442bb86d966ae9e97cbc6 to start an election
I20240502 01:58:18.188396 31307 raft_consensus.cc:1067] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Signalling peer 0773aa6e61f442bb86d966ae9e97cbc6 to start an election
I20240502 01:58:18.188905 30135 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "43d194188c6f455aab0099df8ed0cbbc"
dest_uuid: "0773aa6e61f442bb86d966ae9e97cbc6"
from {username='slave'} at 127.27.111.195:50785
I20240502 01:58:18.189106 30135 raft_consensus.cc:482] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:18.189216 30135 raft_consensus.cc:3037] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:18.190109 30135 raft_consensus.cc:504] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:18.191232 30135 leader_election.cc:288] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [CANDIDATE]: Term 2 election: Requested vote from peers ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695), 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797), 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198:43065)
I20240502 01:58:18.191548 28543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d194188c6f455aab0099df8ed0cbbc" candidate_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ef3bc5c942d04bd29540a80415308353"
I20240502 01:58:18.191684 28543 raft_consensus.cc:3032] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:18.191735 28543 raft_consensus.cc:729] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ef3bc5c942d04bd29540a80415308353, State: Running, Role: LEADER
I20240502 01:58:18.191895 28543 consensus_queue.cc:260] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:18.192137 28543 raft_consensus.cc:3037] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:18.193279 28543 raft_consensus.cc:2454] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0773aa6e61f442bb86d966ae9e97cbc6 in term 2.
I20240502 01:58:18.195284 28674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d194188c6f455aab0099df8ed0cbbc" candidate_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8cb8626af432419c922ceefff85f69f2"
I20240502 01:58:18.195462 28674 raft_consensus.cc:3037] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:18.196216 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d194188c6f455aab0099df8ed0cbbc" candidate_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "34a8610adef8401085b62d8fb1fbc8fe"
I20240502 01:58:18.196405 29873 raft_consensus.cc:3037] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:18.196403 28674 raft_consensus.cc:2454] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0773aa6e61f442bb86d966ae9e97cbc6 in term 2.
I20240502 01:58:18.196841 30071 leader_election.cc:302] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0773aa6e61f442bb86d966ae9e97cbc6, 8cb8626af432419c922ceefff85f69f2, ef3bc5c942d04bd29540a80415308353; no voters:
I20240502 01:58:18.197219 31312 raft_consensus.cc:2781] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:18.197386 29873 raft_consensus.cc:2454] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0773aa6e61f442bb86d966ae9e97cbc6 in term 2.
I20240502 01:58:18.197465 31312 raft_consensus.cc:686] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 LEADER]: Becoming Leader. State: Replica: 0773aa6e61f442bb86d966ae9e97cbc6, State: Running, Role: LEADER
I20240502 01:58:18.197777 31312 consensus_queue.cc:237] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:18.200503 28136 catalog_manager.cc:5484] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 reported cstate change: term changed from 1 to 2, leader changed from ef3bc5c942d04bd29540a80415308353 (127.27.111.195) to 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200). New cstate: current_term: 2 leader_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:58:18.301909 28345 consensus_peers.cc:485] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:58:18.449596 31420 ts_tablet_manager.cc:902] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033: Initiating tablet copy from peer 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369)
I20240502 01:58:18.450165 31420 tablet_copy_client.cc:308] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:35369
I20240502 01:58:18.453931 28432 tablet_copy_service.cc:140] P 8f4edf52bb2d4624ae596b032e50c2a4: Received BeginTabletCopySession request for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c from peer a1a0ca3ab1ca4872b2bb4395068b7033 ({username='slave'} at 127.27.111.197:32843)
I20240502 01:58:18.454078 28432 tablet_copy_service.cc:161] P 8f4edf52bb2d4624ae596b032e50c2a4: Beginning new tablet copy session on tablet 5e87ab8d205e43098dd84ae5f4eeeb6c from peer a1a0ca3ab1ca4872b2bb4395068b7033 at {username='slave'} at 127.27.111.197:32843: session id = a1a0ca3ab1ca4872b2bb4395068b7033-5e87ab8d205e43098dd84ae5f4eeeb6c
I20240502 01:58:18.454949 28432 tablet_copy_source_session.cc:205] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:18.455834 31420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:18.457739 31420 tablet_copy_client.cc:783] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:18.457880 31420 tablet_copy_client.cc:647] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:18.459213 31424 tablet_copy_client.cc:1002] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033: tablet copy: Injecting latency into file download: 1500
I20240502 01:58:18.571278 31426 ts_tablet_manager.cc:902] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e: Initiating tablet copy from peer 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369)
I20240502 01:58:18.571885 31426 tablet_copy_client.cc:308] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:35369
I20240502 01:58:18.572463 28432 tablet_copy_service.cc:140] P 8f4edf52bb2d4624ae596b032e50c2a4: Received BeginTabletCopySession request for tablet 14037ba64754436ba114928a3e0dcbe0 from peer dd5f234ed11745168cd67ee403ff098e ({username='slave'} at 127.27.111.199:33697)
I20240502 01:58:18.572595 28432 tablet_copy_service.cc:161] P 8f4edf52bb2d4624ae596b032e50c2a4: Beginning new tablet copy session on tablet 14037ba64754436ba114928a3e0dcbe0 from peer dd5f234ed11745168cd67ee403ff098e at {username='slave'} at 127.27.111.199:33697: session id = dd5f234ed11745168cd67ee403ff098e-14037ba64754436ba114928a3e0dcbe0
I20240502 01:58:18.573626 28432 tablet_copy_source_session.cc:205] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:18.574725 31426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14037ba64754436ba114928a3e0dcbe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:18.576670 31426 tablet_copy_client.cc:783] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:18.576841 31426 tablet_copy_client.cc:647] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:18.577875 31430 tablet_copy_client.cc:1002] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e: tablet copy: Injecting latency into file download: 1500
I20240502 01:58:18.753829 28543 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 2 FOLLOWER]: Refusing update from remote peer 0773aa6e61f442bb86d966ae9e97cbc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:18.754426 31446 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:18.778185 29873 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Refusing update from remote peer 0773aa6e61f442bb86d966ae9e97cbc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:18.778666 31446 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:18.781917 28674 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Refusing update from remote peer 0773aa6e61f442bb86d966ae9e97cbc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:18.783061 31449 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:19.202548 30135 consensus_queue.cc:237] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:19.203688 28674 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Refusing update from remote peer 0773aa6e61f442bb86d966ae9e97cbc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:19.203727 29873 raft_consensus.cc:1261] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Refusing update from remote peer 0773aa6e61f442bb86d966ae9e97cbc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:19.204064 31451 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:19.204288 31449 consensus_queue.cc:1035] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:19.205734 31446 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ef3bc5c942d04bd29540a80415308353 (127.27.111.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:19.206153 29873 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ef3bc5c942d04bd29540a80415308353 (127.27.111.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:19.206246 28674 raft_consensus.cc:2932] T 43d194188c6f455aab0099df8ed0cbbc P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ef3bc5c942d04bd29540a80415308353 (127.27.111.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:19.207912 28124 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 43d194188c6f455aab0099df8ed0cbbc with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:19.208647 28136 catalog_manager.cc:5484] T 43d194188c6f455aab0099df8ed0cbbc P 0773aa6e61f442bb86d966ae9e97cbc6 reported cstate change: config changed from index 8 to 10, VOTER ef3bc5c942d04bd29540a80415308353 (127.27.111.195) evicted. New cstate: current_term: 2 leader_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:58:19.211704 28476 consensus_peers.cc:485] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:58:19.212054 28523 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d194188c6f455aab0099df8ed0cbbc with delete_type TABLET_DATA_TOMBSTONED (TS ef3bc5c942d04bd29540a80415308353 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51302
I20240502 01:58:19.212381 31494 tablet_replica.cc:330] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353: stopping tablet replica
I20240502 01:58:19.212510 31494 raft_consensus.cc:2229] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:19.212613 31494 raft_consensus.cc:2258] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:19.213232 31494 ts_tablet_manager.cc:1875] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:19.214977 31494 ts_tablet_manager.cc:1888] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:58:19.215040 31494 log.cc:1196] T 43d194188c6f455aab0099df8ed0cbbc P ef3bc5c942d04bd29540a80415308353: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/43d194188c6f455aab0099df8ed0cbbc
I20240502 01:58:19.215555 28124 catalog_manager.cc:4830] TS ef3bc5c942d04bd29540a80415308353 (127.27.111.195:46695): tablet 43d194188c6f455aab0099df8ed0cbbc (table rebalance_test_table_1 [id=eae4e7ef163a4d2788386ab9c9fd7841]) successfully deleted
W20240502 01:58:19.319018 28607 consensus_peers.cc:485] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:58:19.787094 28607 consensus_peers.cc:485] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:58:19.959506 31420 tablet_copy_client.cc:523] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:19.960793 31420 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033: Bootstrap starting.
I20240502 01:58:19.977952 31420 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:19.978238 31420 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033: Bootstrap complete.
I20240502 01:58:19.978456 31420 ts_tablet_manager.cc:1367] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033: Time spent bootstrapping tablet: real 0.018s user 0.019s sys 0.000s
I20240502 01:58:19.978992 31420 raft_consensus.cc:348] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: true } }
I20240502 01:58:19.979104 31420 raft_consensus.cc:729] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a1a0ca3ab1ca4872b2bb4395068b7033, State: Initialized, Role: LEARNER
I20240502 01:58:19.979197 31420 consensus_queue.cc:260] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: true } }
I20240502 01:58:19.979943 31420 ts_tablet_manager.cc:1398] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:19.980469 28432 tablet_copy_service.cc:342] P 8f4edf52bb2d4624ae596b032e50c2a4: Request end of tablet copy session a1a0ca3ab1ca4872b2bb4395068b7033-5e87ab8d205e43098dd84ae5f4eeeb6c received from {username='slave'} at 127.27.111.197:32843
I20240502 01:58:19.980571 28432 tablet_copy_service.cc:433] P 8f4edf52bb2d4624ae596b032e50c2a4: ending tablet copy session a1a0ca3ab1ca4872b2bb4395068b7033-5e87ab8d205e43098dd84ae5f4eeeb6c on tablet 5e87ab8d205e43098dd84ae5f4eeeb6c with peer a1a0ca3ab1ca4872b2bb4395068b7033
W20240502 01:58:20.027532 28607 consensus_peers.cc:485] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:58:20.078233 31426 tablet_copy_client.cc:523] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:20.079643 31426 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e: Bootstrap starting.
I20240502 01:58:20.098098 31426 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:20.098383 31426 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e: Bootstrap complete.
I20240502 01:58:20.098590 31426 ts_tablet_manager.cc:1367] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.005s
I20240502 01:58:20.099038 31426 raft_consensus.cc:348] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } }
I20240502 01:58:20.099164 31426 raft_consensus.cc:729] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: dd5f234ed11745168cd67ee403ff098e, State: Initialized, Role: LEARNER
I20240502 01:58:20.099251 31426 consensus_queue.cc:260] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: NON_VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: true } }
I20240502 01:58:20.099561 31426 ts_tablet_manager.cc:1398] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:20.100148 28432 tablet_copy_service.cc:342] P 8f4edf52bb2d4624ae596b032e50c2a4: Request end of tablet copy session dd5f234ed11745168cd67ee403ff098e-14037ba64754436ba114928a3e0dcbe0 received from {username='slave'} at 127.27.111.199:33697
I20240502 01:58:20.100239 28432 tablet_copy_service.cc:433] P 8f4edf52bb2d4624ae596b032e50c2a4: ending tablet copy session dd5f234ed11745168cd67ee403ff098e-14037ba64754436ba114928a3e0dcbe0 on tablet 14037ba64754436ba114928a3e0dcbe0 with peer dd5f234ed11745168cd67ee403ff098e
W20240502 01:58:20.146188 28345 consensus_peers.cc:485] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:58:20.197192 28607 consensus_peers.cc:485] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:58:20.228060 28476 consensus_peers.cc:485] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:58:20.241174 28345 consensus_peers.cc:485] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:58:20.338531 28345 consensus_peers.cc:485] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:58:20.346302 29742 raft_consensus.cc:1203] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:58:20.460391 30004 raft_consensus.cc:1203] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20240502 01:58:20.701227 28476 consensus_peers.cc:485] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:58:20.787279 28345 consensus_peers.cc:485] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:58:20.823752 31595 raft_consensus.cc:1050] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4: attempting to promote NON_VOTER a1a0ca3ab1ca4872b2bb4395068b7033 to VOTER
I20240502 01:58:20.824151 31595 consensus_queue.cc:237] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } }
I20240502 01:58:20.825551 29742 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 1 LEARNER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:20.825573 28674 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:20.825675 30004 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 1 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:58:20.825951 31597 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:20.826143 31473 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:20.826380 31595 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:58:20.828202 31597 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:20.828610 28674 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:20.828629 30004 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:20.830879 28130 catalog_manager.cc:5484] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 reported cstate change: config changed from index 7 to 8, a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:20.832042 29742 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:21.004597 31473 raft_consensus.cc:1050] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: attempting to promote NON_VOTER dd5f234ed11745168cd67ee403ff098e to VOTER
I20240502 01:58:21.005024 31473 consensus_queue.cc:237] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } }
I20240502 01:58:21.006513 29873 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:21.006551 30004 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 2 LEARNER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:21.006567 28674 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:21.006956 31595 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:21.007159 31602 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:21.007411 31595 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:21.009399 31602 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:21.009949 28674 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:21.011238 30004 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:21.012405 28136 catalog_manager.cc:5484] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 reported cstate change: config changed from index 9 to 10, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:21.013549 29873 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, dd5f234ed11745168cd67ee403ff098e (127.27.111.199) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:21.016970 28411 consensus_queue.cc:237] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } }
I20240502 01:58:21.018306 29873 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:58:21.018705 31597 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:58:21.018986 30004 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:58:21.019342 31597 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:58:21.020969 31473 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 8cb8626af432419c922ceefff85f69f2 (127.27.111.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:21.022734 29872 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 8cb8626af432419c922ceefff85f69f2 (127.27.111.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:21.022995 28122 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 14037ba64754436ba114928a3e0dcbe0 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:21.023146 28411 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c"
dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39516
I20240502 01:58:21.023286 28411 raft_consensus.cc:595] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:21.024927 28136 catalog_manager.cc:5484] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe reported cstate change: config changed from index 10 to 11, VOTER 8cb8626af432419c922ceefff85f69f2 (127.27.111.196) evicted. New cstate: current_term: 2 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:21.025147 30004 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 8cb8626af432419c922ceefff85f69f2 (127.27.111.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } }
I20240502 01:58:21.028175 28654 tablet_service.cc:1514] Processing DeleteTablet for tablet 14037ba64754436ba114928a3e0dcbe0 with delete_type TABLET_DATA_TOMBSTONED (TS 8cb8626af432419c922ceefff85f69f2 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:49092
I20240502 01:58:21.028646 31626 tablet_replica.cc:330] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2: stopping tablet replica
I20240502 01:58:21.028986 31626 raft_consensus.cc:2229] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:21.029237 31626 raft_consensus.cc:2258] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:21.030508 31626 ts_tablet_manager.cc:1875] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:21.032917 31626 ts_tablet_manager.cc:1888] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 01:58:21.033010 31626 log.cc:1196] T 14037ba64754436ba114928a3e0dcbe0 P 8cb8626af432419c922ceefff85f69f2: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/14037ba64754436ba114928a3e0dcbe0
I20240502 01:58:21.033609 28125 catalog_manager.cc:4830] TS 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797): tablet 14037ba64754436ba114928a3e0dcbe0 (table rebalance_test_table_4 [id=6a3a04e2234e47ff9c73333582341e02]) successfully deleted
I20240502 01:58:21.041088 28411 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c"
dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39544
I20240502 01:58:21.041211 28411 raft_consensus.cc:595] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:21.223402 31584 raft_consensus.cc:979] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4: : Instructing follower 8cb8626af432419c922ceefff85f69f2 to start an election
I20240502 01:58:21.223523 31584 raft_consensus.cc:1067] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Signalling peer 8cb8626af432419c922ceefff85f69f2 to start an election
I20240502 01:58:21.224036 28674 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c"
dest_uuid: "8cb8626af432419c922ceefff85f69f2"
from {username='slave'} at 127.27.111.194:33373
I20240502 01:58:21.224172 28674 raft_consensus.cc:482] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:21.224236 28674 raft_consensus.cc:3037] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:21.225157 28674 raft_consensus.cc:504] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } }
I20240502 01:58:21.225625 28674 leader_election.cc:288] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 2 election: Requested vote from peers 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369), dd5f234ed11745168cd67ee403ff098e (127.27.111.199:42203), a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197:40085)
I20240502 01:58:21.225929 28411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
I20240502 01:58:21.226049 28411 raft_consensus.cc:3032] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:21.226042 29742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033"
I20240502 01:58:21.226096 28411 raft_consensus.cc:729] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Running, Role: LEADER
I20240502 01:58:21.226161 29742 raft_consensus.cc:3037] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:21.226155 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c" candidate_uuid: "8cb8626af432419c922ceefff85f69f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "dd5f234ed11745168cd67ee403ff098e"
I20240502 01:58:21.226284 28411 consensus_queue.cc:260] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } }
I20240502 01:58:21.226349 30004 raft_consensus.cc:3037] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:21.226533 28411 raft_consensus.cc:3037] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:21.227247 30004 raft_consensus.cc:2454] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 2.
I20240502 01:58:21.227283 29742 raft_consensus.cc:2454] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 2.
I20240502 01:58:21.227545 28411 raft_consensus.cc:2454] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cb8626af432419c922ceefff85f69f2 in term 2.
I20240502 01:58:21.227609 28610 leader_election.cc:302] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, a1a0ca3ab1ca4872b2bb4395068b7033, dd5f234ed11745168cd67ee403ff098e; no voters:
I20240502 01:58:21.227790 31320 raft_consensus.cc:2781] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:21.227944 31320 raft_consensus.cc:686] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Becoming Leader. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Running, Role: LEADER
I20240502 01:58:21.228142 31320 consensus_queue.cc:237] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } }
I20240502 01:58:21.229990 28136 catalog_manager.cc:5484] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 reported cstate change: term changed from 1 to 2, leader changed from 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) to 8cb8626af432419c922ceefff85f69f2 (127.27.111.196). New cstate: current_term: 2 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:21.468494 28607 consensus_peers.cc:485] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:58:21.533793 28476 consensus_peers.cc:485] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:58:21.713065 29742 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:21.713505 31637 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:21.723001 28411 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:21.723358 31637 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:58:21.725829 30004 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:58:21.726442 31320 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 01:58:22.199858 28607 consensus_peers.cc:485] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:58:22.248955 28674 consensus_queue.cc:237] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } }
I20240502 01:58:22.250116 29742 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:22.250098 30004 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:58:22.250509 31637 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:22.250708 31672 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:58:22.252396 31637 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:22.252818 29742 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:22.253073 30004 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } }
I20240502 01:58:22.254678 28125 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:22.255095 28130 catalog_manager.cc:5484] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 reported cstate change: config changed from index 8 to 10, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New cstate: current_term: 2 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:22.257907 28391 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c with delete_type TABLET_DATA_TOMBSTONED (TS 8f4edf52bb2d4624ae596b032e50c2a4 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36436
I20240502 01:58:22.258265 31680 tablet_replica.cc:330] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4: stopping tablet replica
I20240502 01:58:22.258435 31680 raft_consensus.cc:2229] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:22.258560 31680 raft_consensus.cc:2258] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:22.259224 31680 ts_tablet_manager.cc:1875] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:22.260995 31680 ts_tablet_manager.cc:1888] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:58:22.261062 31680 log.cc:1196] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8f4edf52bb2d4624ae596b032e50c2a4: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/5e87ab8d205e43098dd84ae5f4eeeb6c
I20240502 01:58:22.261487 28122 catalog_manager.cc:4830] TS 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369): tablet 5e87ab8d205e43098dd84ae5f4eeeb6c (table rebalance_test_table_1 [id=eae4e7ef163a4d2788386ab9c9fd7841]) successfully deleted
I20240502 01:58:22.339710 30115 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:22.341563 28654 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:22.346174 28391 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:22.347051 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:22.351699 29722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:22.362824 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:22.368146 28523 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:22.399433 28674 consensus_queue.cc:237] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } }
I20240502 01:58:22.400959 30004 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:58:22.401486 29742 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:58:22.401497 31673 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:58:22.402102 31673 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:58:22.403482 31673 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } } }
W20240502 01:58:22.404656 28609 consensus_peers.cc:485] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 -> Peer 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200:39123): Couldn't send request to peer 0773aa6e61f442bb86d966ae9e97cbc6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30179e5d45eb47cd88045b81e4872c78. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:22.405982 30004 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } } }
I20240502 01:58:22.406093 28130 catalog_manager.cc:5484] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 reported cstate change: config changed from index 10 to 11, NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) added. New cstate: current_term: 2 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:22.409797 29742 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } } }
W20240502 01:58:22.570931 28607 consensus_peers.cc:485] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:58:22.677978 28345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 01:58:22.678771 28345 consensus_peers.cc:485] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:58:22.679204 28345 consensus_peers.cc:485] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:58:22.749269 28607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 01:58:22.749997 28607 consensus_peers.cc:485] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:58:22.759582 28476 consensus_peers.cc:485] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:58:22.954530 31750 ts_tablet_manager.cc:902] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6: Initiating tablet copy from peer 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:22.955005 31750 tablet_copy_client.cc:308] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:40797
I20240502 01:58:22.955476 28694 tablet_copy_service.cc:140] P 8cb8626af432419c922ceefff85f69f2: Received BeginTabletCopySession request for tablet 30179e5d45eb47cd88045b81e4872c78 from peer 0773aa6e61f442bb86d966ae9e97cbc6 ({username='slave'} at 127.27.111.200:57553)
I20240502 01:58:22.955590 28694 tablet_copy_service.cc:161] P 8cb8626af432419c922ceefff85f69f2: Beginning new tablet copy session on tablet 30179e5d45eb47cd88045b81e4872c78 from peer 0773aa6e61f442bb86d966ae9e97cbc6 at {username='slave'} at 127.27.111.200:57553: session id = 0773aa6e61f442bb86d966ae9e97cbc6-30179e5d45eb47cd88045b81e4872c78
I20240502 01:58:22.956323 28694 tablet_copy_source_session.cc:205] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:22.957167 31750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30179e5d45eb47cd88045b81e4872c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:22.959084 31750 tablet_copy_client.cc:783] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:22.959242 31750 tablet_copy_client.cc:647] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:22.960227 31754 tablet_copy_client.cc:1002] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Injecting latency into file download: 1500
W20240502 01:58:22.976083 28476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 01:58:23.004593 28345 consensus_peers.cc:485] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:58:23.166322 28345 consensus_peers.cc:485] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:58:23.236946 28476 consensus_peers.cc:485] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:58:24.165371 28607 consensus_peers.cc:485] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:58:24.195899 28476 consensus_peers.cc:485] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:58:24.460566 31750 tablet_copy_client.cc:523] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:24.462047 31750 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6: Bootstrap starting.
I20240502 01:58:24.483597 31750 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:24.483886 31750 tablet_bootstrap.cc:492] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6: Bootstrap complete.
I20240502 01:58:24.484031 31750 ts_tablet_manager.cc:1367] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.005s
I20240502 01:58:24.484566 31750 raft_consensus.cc:348] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } }
I20240502 01:58:24.484673 31750 raft_consensus.cc:729] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 0773aa6e61f442bb86d966ae9e97cbc6, State: Initialized, Role: LEARNER
I20240502 01:58:24.484755 31750 consensus_queue.cc:260] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } }
I20240502 01:58:24.485031 31750 ts_tablet_manager.cc:1398] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:24.485585 28694 tablet_copy_service.cc:342] P 8cb8626af432419c922ceefff85f69f2: Request end of tablet copy session 0773aa6e61f442bb86d966ae9e97cbc6-30179e5d45eb47cd88045b81e4872c78 received from {username='slave'} at 127.27.111.200:57553
I20240502 01:58:24.485682 28694 tablet_copy_service.cc:433] P 8cb8626af432419c922ceefff85f69f2: ending tablet copy session 0773aa6e61f442bb86d966ae9e97cbc6-30179e5d45eb47cd88045b81e4872c78 on tablet 30179e5d45eb47cd88045b81e4872c78 with peer 0773aa6e61f442bb86d966ae9e97cbc6
W20240502 01:58:24.557583 28607 consensus_peers.cc:485] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:58:24.589324 30135 raft_consensus.cc:1203] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
W20240502 01:58:24.989194 28607 consensus_peers.cc:485] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:58:25.009094 31811 raft_consensus.cc:1050] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2: attempting to promote NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 to VOTER
I20240502 01:58:25.009548 31811 consensus_queue.cc:237] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:25.010835 30135 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 LEARNER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:58:25.010936 30004 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:58:25.011027 29742 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:58:25.011329 31811 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:58:25.011591 31810 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:58:25.012434 31813 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:58:25.014216 31811 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:25.014701 30004 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:25.015259 30135 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:25.017602 28136 catalog_manager.cc:5484] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6 reported cstate change: config changed from index 11 to 12, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:25.018368 29742 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:25.105430 28674 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78"
dest_uuid: "8cb8626af432419c922ceefff85f69f2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50276
I20240502 01:58:25.105602 28674 raft_consensus.cc:595] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Received request to transfer leadership
W20240502 01:58:25.159637 28345 consensus_peers.cc:485] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:58:25.241814 28476 consensus_peers.cc:485] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:58:25.284085 28345 consensus_peers.cc:485] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:58:25.317118 28674 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78"
dest_uuid: "8cb8626af432419c922ceefff85f69f2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50306
I20240502 01:58:25.317270 28674 raft_consensus.cc:595] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Received request to transfer leadership
W20240502 01:58:25.434824 28476 consensus_peers.cc:485] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:58:25.437813 31810 raft_consensus.cc:979] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2: : Instructing follower a1a0ca3ab1ca4872b2bb4395068b7033 to start an election
I20240502 01:58:25.437944 31810 raft_consensus.cc:1067] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Signalling peer a1a0ca3ab1ca4872b2bb4395068b7033 to start an election
I20240502 01:58:25.438565 29742 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78"
dest_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033"
from {username='slave'} at 127.27.111.196:38517
I20240502 01:58:25.438742 29742 raft_consensus.cc:482] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:25.438799 29742 raft_consensus.cc:3037] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:25.439617 29742 raft_consensus.cc:504] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:25.440686 29742 leader_election.cc:288] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [CANDIDATE]: Term 3 election: Requested vote from peers 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797), dd5f234ed11745168cd67ee403ff098e (127.27.111.199:42203), 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200:39123)
I20240502 01:58:25.445055 28674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78" candidate_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "8cb8626af432419c922ceefff85f69f2"
I20240502 01:58:25.445219 28674 raft_consensus.cc:3032] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:58:25.445269 28674 raft_consensus.cc:729] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Running, Role: LEADER
I20240502 01:58:25.445235 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78" candidate_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "0773aa6e61f442bb86d966ae9e97cbc6"
I20240502 01:58:25.445276 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30179e5d45eb47cd88045b81e4872c78" candidate_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "dd5f234ed11745168cd67ee403ff098e"
I20240502 01:58:25.445396 30004 raft_consensus.cc:3037] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:25.445446 30135 raft_consensus.cc:3037] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:25.445461 28674 consensus_queue.cc:260] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:25.445715 28674 raft_consensus.cc:3037] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:25.446450 30004 raft_consensus.cc:2454] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a0ca3ab1ca4872b2bb4395068b7033 in term 3.
I20240502 01:58:25.446457 30135 raft_consensus.cc:2454] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a0ca3ab1ca4872b2bb4395068b7033 in term 3.
I20240502 01:58:25.446527 28674 raft_consensus.cc:2454] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a0ca3ab1ca4872b2bb4395068b7033 in term 3.
I20240502 01:58:25.446918 29678 leader_election.cc:302] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0773aa6e61f442bb86d966ae9e97cbc6, a1a0ca3ab1ca4872b2bb4395068b7033, dd5f234ed11745168cd67ee403ff098e; no voters:
I20240502 01:58:25.447171 31822 raft_consensus.cc:2781] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:58:25.447403 31822 raft_consensus.cc:686] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 3 LEADER]: Becoming Leader. State: Replica: a1a0ca3ab1ca4872b2bb4395068b7033, State: Running, Role: LEADER
I20240502 01:58:25.447746 31822 consensus_queue.cc:237] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:25.450275 28136 catalog_manager.cc:5484] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 reported cstate change: term changed from 2 to 3, leader changed from 8cb8626af432419c922ceefff85f69f2 (127.27.111.196) to a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197). New cstate: current_term: 3 leader_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 01:58:25.469153 28607 consensus_peers.cc:485] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:58:25.550549 28345 consensus_peers.cc:485] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:58:25.590777 28345 consensus_peers.cc:485] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:58:25.837997 30004 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 3 FOLLOWER]: Refusing update from remote peer a1a0ca3ab1ca4872b2bb4395068b7033: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 01:58:25.837997 30135 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 3 FOLLOWER]: Refusing update from remote peer a1a0ca3ab1ca4872b2bb4395068b7033: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 01:58:25.838555 31848 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:58:25.838768 31822 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:58:25.842754 28674 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 3 FOLLOWER]: Refusing update from remote peer a1a0ca3ab1ca4872b2bb4395068b7033: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 01:58:25.843307 31848 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:58:25.849213 29742 consensus_queue.cc:237] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:25.850282 30135 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 3 FOLLOWER]: Refusing update from remote peer a1a0ca3ab1ca4872b2bb4395068b7033: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 01:58:25.850308 30004 raft_consensus.cc:1261] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 3 FOLLOWER]: Refusing update from remote peer a1a0ca3ab1ca4872b2bb4395068b7033: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 01:58:25.850672 31848 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:58:25.850809 31862 consensus_queue.cc:1035] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:58:25.852557 31822 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 8cb8626af432419c922ceefff85f69f2 (127.27.111.196) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:25.853066 30135 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 8cb8626af432419c922ceefff85f69f2 (127.27.111.196) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:25.853067 30004 raft_consensus.cc:2932] T 30179e5d45eb47cd88045b81e4872c78 P dd5f234ed11745168cd67ee403ff098e [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 8cb8626af432419c922ceefff85f69f2 (127.27.111.196) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:25.854521 28125 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 30179e5d45eb47cd88045b81e4872c78 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:25.855481 28136 catalog_manager.cc:5484] T 30179e5d45eb47cd88045b81e4872c78 P a1a0ca3ab1ca4872b2bb4395068b7033 reported cstate change: config changed from index 12 to 14, VOTER 8cb8626af432419c922ceefff85f69f2 (127.27.111.196) evicted. New cstate: current_term: 3 leader_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:25.858134 28654 tablet_service.cc:1514] Processing DeleteTablet for tablet 30179e5d45eb47cd88045b81e4872c78 with delete_type TABLET_DATA_TOMBSTONED (TS 8cb8626af432419c922ceefff85f69f2 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:49092
I20240502 01:58:25.858557 31867 tablet_replica.cc:330] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2: stopping tablet replica
I20240502 01:58:25.858681 31867 raft_consensus.cc:2229] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:25.858776 31867 raft_consensus.cc:2258] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:25.859418 31867 ts_tablet_manager.cc:1875] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:25.861596 31867 ts_tablet_manager.cc:1888] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240502 01:58:25.861673 31867 log.cc:1196] T 30179e5d45eb47cd88045b81e4872c78 P 8cb8626af432419c922ceefff85f69f2: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/30179e5d45eb47cd88045b81e4872c78
I20240502 01:58:25.862201 28125 catalog_manager.cc:4830] TS 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797): tablet 30179e5d45eb47cd88045b81e4872c78 (table rebalance_test_table_3 [id=86cf304e1bdb4ceabae33570f6c0d7fc]) successfully deleted
I20240502 01:58:25.946240 28411 consensus_queue.cc:237] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } }
I20240502 01:58:25.947938 29873 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:58:25.948268 30004 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:58:25.948391 31794 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:58:25.948737 31794 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:58:25.950675 31880 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } } }
I20240502 01:58:25.951151 29873 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } } }
W20240502 01:58:25.952646 28347 consensus_peers.cc:485] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200:39123): Couldn't send request to peer 0773aa6e61f442bb86d966ae9e97cbc6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14037ba64754436ba114928a3e0dcbe0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:25.954160 30004 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } } }
I20240502 01:58:25.954156 28130 catalog_manager.cc:5484] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 reported cstate change: config changed from index 11 to 12, NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) added. New cstate: current_term: 2 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:25.962510 28674 consensus_queue.cc:237] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:25.964083 30004 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:58:25.964186 29742 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:58:25.964681 31893 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:58:25.964936 31810 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:58:25.966553 31894 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } } }
I20240502 01:58:25.966949 30004 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } } }
I20240502 01:58:25.967424 29742 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } } }
W20240502 01:58:25.969657 28607 consensus_peers.cc:485] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 -> Peer 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198:43065): Couldn't send request to peer 34a8610adef8401085b62d8fb1fbc8fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e87ab8d205e43098dd84ae5f4eeeb6c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:25.969596 28136 catalog_manager.cc:5484] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e reported cstate change: config changed from index 10 to 11, NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) added. New cstate: current_term: 2 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } } }
W20240502 01:58:26.047750 28122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
W20240502 01:58:26.048730 28122 catalog_manager.cc:4573] TS a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 14037ba64754436ba114928a3e0dcbe0: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111)
I20240502 01:58:26.386268 31919 ts_tablet_manager.cc:902] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6: Initiating tablet copy from peer 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369)
I20240502 01:58:26.386787 31919 tablet_copy_client.cc:308] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:35369
I20240502 01:58:26.390089 28432 tablet_copy_service.cc:140] P 8f4edf52bb2d4624ae596b032e50c2a4: Received BeginTabletCopySession request for tablet 14037ba64754436ba114928a3e0dcbe0 from peer 0773aa6e61f442bb86d966ae9e97cbc6 ({username='slave'} at 127.27.111.200:48563)
I20240502 01:58:26.390192 28432 tablet_copy_service.cc:161] P 8f4edf52bb2d4624ae596b032e50c2a4: Beginning new tablet copy session on tablet 14037ba64754436ba114928a3e0dcbe0 from peer 0773aa6e61f442bb86d966ae9e97cbc6 at {username='slave'} at 127.27.111.200:48563: session id = 0773aa6e61f442bb86d966ae9e97cbc6-14037ba64754436ba114928a3e0dcbe0
I20240502 01:58:26.391114 28432 tablet_copy_source_session.cc:205] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:26.391955 31919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14037ba64754436ba114928a3e0dcbe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:26.393759 31919 tablet_copy_client.cc:783] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:26.393926 31919 tablet_copy_client.cc:647] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:26.394992 31923 tablet_copy_client.cc:1002] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Injecting latency into file download: 1500
I20240502 01:58:26.520107 31935 ts_tablet_manager.cc:902] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe: Initiating tablet copy from peer 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797)
I20240502 01:58:26.520612 31935 tablet_copy_client.cc:308] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:40797
I20240502 01:58:26.524189 28694 tablet_copy_service.cc:140] P 8cb8626af432419c922ceefff85f69f2: Received BeginTabletCopySession request for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c from peer 34a8610adef8401085b62d8fb1fbc8fe ({username='slave'} at 127.27.111.198:56905)
I20240502 01:58:26.524305 28694 tablet_copy_service.cc:161] P 8cb8626af432419c922ceefff85f69f2: Beginning new tablet copy session on tablet 5e87ab8d205e43098dd84ae5f4eeeb6c from peer 34a8610adef8401085b62d8fb1fbc8fe at {username='slave'} at 127.27.111.198:56905: session id = 34a8610adef8401085b62d8fb1fbc8fe-5e87ab8d205e43098dd84ae5f4eeeb6c
I20240502 01:58:26.525094 28694 tablet_copy_source_session.cc:205] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:26.526016 31935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:26.527972 31935 tablet_copy_client.cc:783] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:26.528120 31935 tablet_copy_client.cc:647] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:26.529222 31939 tablet_copy_client.cc:1002] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Injecting latency into file download: 1500
W20240502 01:58:26.602311 28607 consensus_peers.cc:485] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:58:26.622449 28476 consensus_peers.cc:485] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:58:27.044323 28607 consensus_peers.cc:485] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:58:27.562203 28345 consensus_peers.cc:485] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:58:27.650846 28607 consensus_peers.cc:485] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:58:27.716272 28345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 01:58:27.716984 28345 consensus_peers.cc:485] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:58:27.810079 28607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 01:58:27.810700 28607 consensus_peers.cc:485] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:58:27.850126 28345 consensus_peers.cc:485] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:58:27.895327 31919 tablet_copy_client.cc:523] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:27.896736 31919 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6: Bootstrap starting.
I20240502 01:58:27.917940 31919 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:27.918244 31919 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6: Bootstrap complete.
I20240502 01:58:27.918452 31919 ts_tablet_manager.cc:1367] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.004s
I20240502 01:58:27.918850 31919 raft_consensus.cc:348] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } }
I20240502 01:58:27.918985 31919 raft_consensus.cc:729] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 0773aa6e61f442bb86d966ae9e97cbc6, State: Initialized, Role: LEARNER
I20240502 01:58:27.919099 31919 consensus_queue.cc:260] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: NON_VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: true } }
I20240502 01:58:27.919414 31919 ts_tablet_manager.cc:1398] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:27.919924 28432 tablet_copy_service.cc:342] P 8f4edf52bb2d4624ae596b032e50c2a4: Request end of tablet copy session 0773aa6e61f442bb86d966ae9e97cbc6-14037ba64754436ba114928a3e0dcbe0 received from {username='slave'} at 127.27.111.200:48563
I20240502 01:58:27.920009 28432 tablet_copy_service.cc:433] P 8f4edf52bb2d4624ae596b032e50c2a4: ending tablet copy session 0773aa6e61f442bb86d966ae9e97cbc6-14037ba64754436ba114928a3e0dcbe0 on tablet 14037ba64754436ba114928a3e0dcbe0 with peer 0773aa6e61f442bb86d966ae9e97cbc6
W20240502 01:58:27.964161 28476 consensus_peers.cc:485] T 4f816e20fed94a0587d9c7cfac7071cc P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:58:28.000639 28476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 01:58:28.001299 28476 consensus_peers.cc:485] T cd3a11e1ac624f3fbc8cacf351b8afa5 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:58:28.026997 30135 raft_consensus.cc:1203] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240502 01:58:28.029548 31935 tablet_copy_client.cc:523] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:28.031014 31935 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe: Bootstrap starting.
W20240502 01:58:28.047403 28345 consensus_peers.cc:485] T fa11ebedbf5c4038a5388d9440688011 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:58:28.050186 31935 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:28.050489 31935 tablet_bootstrap.cc:492] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe: Bootstrap complete.
I20240502 01:58:28.050647 31935 ts_tablet_manager.cc:1367] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.007s
I20240502 01:58:28.051031 31935 raft_consensus.cc:348] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:28.051141 31935 raft_consensus.cc:729] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 34a8610adef8401085b62d8fb1fbc8fe, State: Initialized, Role: LEARNER
I20240502 01:58:28.051211 31935 consensus_queue.cc:260] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:28.051488 31935 ts_tablet_manager.cc:1398] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:28.052054 28694 tablet_copy_service.cc:342] P 8cb8626af432419c922ceefff85f69f2: Request end of tablet copy session 34a8610adef8401085b62d8fb1fbc8fe-5e87ab8d205e43098dd84ae5f4eeeb6c received from {username='slave'} at 127.27.111.198:56905
I20240502 01:58:28.052139 28694 tablet_copy_service.cc:433] P 8cb8626af432419c922ceefff85f69f2: ending tablet copy session 34a8610adef8401085b62d8fb1fbc8fe-5e87ab8d205e43098dd84ae5f4eeeb6c on tablet 5e87ab8d205e43098dd84ae5f4eeeb6c with peer 34a8610adef8401085b62d8fb1fbc8fe
I20240502 01:58:28.472273 29873 raft_consensus.cc:1203] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20240502 01:58:28.475296 32041 raft_consensus.cc:1050] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: attempting to promote NON_VOTER 0773aa6e61f442bb86d966ae9e97cbc6 to VOTER
I20240502 01:58:28.475695 32041 consensus_queue.cc:237] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:28.477051 29873 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:58:28.477043 30135 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 LEARNER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:58:28.477145 30004 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Refusing update from remote peer 8f4edf52bb2d4624ae596b032e50c2a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:58:28.477535 32041 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:58:28.477711 31986 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:58:28.477910 32044 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:58:28.479820 32044 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:28.480212 30004 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:28.480247 30135 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:28.482167 29873 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:28.483677 28136 catalog_manager.cc:5484] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e reported cstate change: config changed from index 12 to 13, 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:28.539642 28411 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14037ba64754436ba114928a3e0dcbe0"
dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35748
I20240502 01:58:28.539803 28411 raft_consensus.cc:595] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 LEADER]: Received request to transfer leadership
I20240502 01:58:28.757324 28411 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14037ba64754436ba114928a3e0dcbe0"
dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35760
I20240502 01:58:28.757488 28411 raft_consensus.cc:595] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 LEADER]: Received request to transfer leadership
W20240502 01:58:28.784246 28607 consensus_peers.cc:485] T 803991b0b27f470dbfe2397baa997621 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:58:28.872550 32042 raft_consensus.cc:979] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: : Instructing follower 0773aa6e61f442bb86d966ae9e97cbc6 to start an election
I20240502 01:58:28.872668 32042 raft_consensus.cc:1067] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 LEADER]: Signalling peer 0773aa6e61f442bb86d966ae9e97cbc6 to start an election
I20240502 01:58:28.873190 30135 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14037ba64754436ba114928a3e0dcbe0"
dest_uuid: "0773aa6e61f442bb86d966ae9e97cbc6"
from {username='slave'} at 127.27.111.194:46709
I20240502 01:58:28.873330 30135 raft_consensus.cc:482] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:28.873379 30135 raft_consensus.cc:3037] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:28.874343 30135 raft_consensus.cc:504] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:28.874873 30135 leader_election.cc:288] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [CANDIDATE]: Term 3 election: Requested vote from peers 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369), 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198:43065), dd5f234ed11745168cd67ee403ff098e (127.27.111.199:42203)
I20240502 01:58:28.875506 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14037ba64754436ba114928a3e0dcbe0" candidate_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "34a8610adef8401085b62d8fb1fbc8fe"
I20240502 01:58:28.875551 28411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14037ba64754436ba114928a3e0dcbe0" candidate_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "8f4edf52bb2d4624ae596b032e50c2a4"
I20240502 01:58:28.875663 29873 raft_consensus.cc:3037] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:28.875691 28411 raft_consensus.cc:3032] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:58:28.875736 28411 raft_consensus.cc:729] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f4edf52bb2d4624ae596b032e50c2a4, State: Running, Role: LEADER
I20240502 01:58:28.875962 28411 consensus_queue.cc:260] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:28.876184 28411 raft_consensus.cc:3037] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:28.876842 29873 raft_consensus.cc:2454] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0773aa6e61f442bb86d966ae9e97cbc6 in term 3.
I20240502 01:58:28.877278 28411 raft_consensus.cc:2454] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0773aa6e61f442bb86d966ae9e97cbc6 in term 3.
I20240502 01:58:28.877619 30068 leader_election.cc:302] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0773aa6e61f442bb86d966ae9e97cbc6, 34a8610adef8401085b62d8fb1fbc8fe, 8f4edf52bb2d4624ae596b032e50c2a4; no voters:
I20240502 01:58:28.877825 31777 raft_consensus.cc:2781] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:58:28.878015 31777 raft_consensus.cc:686] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 3 LEADER]: Becoming Leader. State: Replica: 0773aa6e61f442bb86d966ae9e97cbc6, State: Running, Role: LEADER
I20240502 01:58:28.878178 31777 consensus_queue.cc:237] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:28.879724 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14037ba64754436ba114928a3e0dcbe0" candidate_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "dd5f234ed11745168cd67ee403ff098e"
I20240502 01:58:28.879835 30004 raft_consensus.cc:3037] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:28.879849 28130 catalog_manager.cc:5484] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 reported cstate change: term changed from 2 to 3, leader changed from 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) to 0773aa6e61f442bb86d966ae9e97cbc6 (127.27.111.200). New cstate: current_term: 3 leader_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:58:28.880628 30004 raft_consensus.cc:2454] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0773aa6e61f442bb86d966ae9e97cbc6 in term 3.
I20240502 01:58:28.887918 32087 raft_consensus.cc:1050] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2: attempting to promote NON_VOTER 34a8610adef8401085b62d8fb1fbc8fe to VOTER
I20240502 01:58:28.888291 32087 consensus_queue.cc:237] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } }
I20240502 01:58:28.889721 29873 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe [term 2 LEARNER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:58:28.889839 30004 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:58:28.890218 29742 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Refusing update from remote peer 8cb8626af432419c922ceefff85f69f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:58:28.890208 32086 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:58:28.890619 32086 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:58:28.890774 32030 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:58:28.892400 32030 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:28.892800 29742 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:28.893785 29873 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:28.894834 30004 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:28.896651 28132 catalog_manager.cc:5484] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe reported cstate change: config changed from index 11 to 12, 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8cb8626af432419c922ceefff85f69f2" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:28.968672 28674 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c"
dest_uuid: "8cb8626af432419c922ceefff85f69f2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50458
I20240502 01:58:28.968818 28674 raft_consensus.cc:595] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Received request to transfer leadership
I20240502 01:58:29.186580 28674 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c"
dest_uuid: "8cb8626af432419c922ceefff85f69f2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50478
I20240502 01:58:29.186720 28674 raft_consensus.cc:595] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Received request to transfer leadership
I20240502 01:58:29.286553 32086 raft_consensus.cc:979] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2: : Instructing follower a1a0ca3ab1ca4872b2bb4395068b7033 to start an election
I20240502 01:58:29.286686 32086 raft_consensus.cc:1067] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Signalling peer a1a0ca3ab1ca4872b2bb4395068b7033 to start an election
I20240502 01:58:29.287143 29742 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c"
dest_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033"
from {username='slave'} at 127.27.111.196:38517
I20240502 01:58:29.287278 29742 raft_consensus.cc:482] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:29.287328 29742 raft_consensus.cc:3037] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:29.288219 29742 raft_consensus.cc:504] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } }
I20240502 01:58:29.288669 29742 leader_election.cc:288] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [CANDIDATE]: Term 3 election: Requested vote from peers 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797), dd5f234ed11745168cd67ee403ff098e (127.27.111.199:42203), 34a8610adef8401085b62d8fb1fbc8fe (127.27.111.198:43065)
I20240502 01:58:29.289077 28674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c" candidate_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "8cb8626af432419c922ceefff85f69f2"
I20240502 01:58:29.289119 30004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c" candidate_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "dd5f234ed11745168cd67ee403ff098e"
I20240502 01:58:29.289211 28674 raft_consensus.cc:3032] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:58:29.289232 30004 raft_consensus.cc:3037] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:29.289278 28674 raft_consensus.cc:729] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8cb8626af432419c922ceefff85f69f2, State: Running, Role: LEADER
I20240502 01:58:29.289456 28674 consensus_queue.cc:260] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } }
I20240502 01:58:29.289664 28674 raft_consensus.cc:3037] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:29.290266 30004 raft_consensus.cc:2454] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a0ca3ab1ca4872b2bb4395068b7033 in term 3.
I20240502 01:58:29.290728 28674 raft_consensus.cc:2454] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a0ca3ab1ca4872b2bb4395068b7033 in term 3.
I20240502 01:58:29.291039 29678 leader_election.cc:302] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8cb8626af432419c922ceefff85f69f2, a1a0ca3ab1ca4872b2bb4395068b7033, dd5f234ed11745168cd67ee403ff098e; no voters:
I20240502 01:58:29.291225 31862 raft_consensus.cc:2781] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:58:29.291402 31862 raft_consensus.cc:686] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 3 LEADER]: Becoming Leader. State: Replica: a1a0ca3ab1ca4872b2bb4395068b7033, State: Running, Role: LEADER
I20240502 01:58:29.291558 31862 consensus_queue.cc:237] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } }
I20240502 01:58:29.293061 29873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e87ab8d205e43098dd84ae5f4eeeb6c" candidate_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "34a8610adef8401085b62d8fb1fbc8fe"
I20240502 01:58:29.293200 29873 raft_consensus.cc:3037] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:58:29.293351 28129 catalog_manager.cc:5484] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 reported cstate change: term changed from 2 to 3, leader changed from 8cb8626af432419c922ceefff85f69f2 (127.27.111.196) to a1a0ca3ab1ca4872b2bb4395068b7033 (127.27.111.197). New cstate: current_term: 3 leader_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:58:29.294173 29873 raft_consensus.cc:2454] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1a0ca3ab1ca4872b2bb4395068b7033 in term 3.
I20240502 01:58:29.302711 29873 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 3 FOLLOWER]: Refusing update from remote peer 0773aa6e61f442bb86d966ae9e97cbc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 01:58:29.303156 31777 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:58:29.320611 30004 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 3 FOLLOWER]: Refusing update from remote peer 0773aa6e61f442bb86d966ae9e97cbc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 01:58:29.321027 31777 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:58:29.324177 28411 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 3 FOLLOWER]: Refusing update from remote peer 0773aa6e61f442bb86d966ae9e97cbc6: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 01:58:29.324627 32085 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:58:29.325862 30135 consensus_queue.cc:237] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } }
I20240502 01:58:29.327186 30004 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 3 FOLLOWER]: Refusing update from remote peer 0773aa6e61f442bb86d966ae9e97cbc6: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240502 01:58:29.327173 29873 raft_consensus.cc:1261] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 3 FOLLOWER]: Refusing update from remote peer 0773aa6e61f442bb86d966ae9e97cbc6: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240502 01:58:29.327533 32127 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 01:58:29.327878 32127 consensus_queue.cc:1035] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 01:58:29.329116 32085 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 0773aa6e61f442bb86d966ae9e97cbc6 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:29.329528 30004 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:29.330675 28124 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 14037ba64754436ba114928a3e0dcbe0 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:29.331883 28129 catalog_manager.cc:5484] T 14037ba64754436ba114928a3e0dcbe0 P dd5f234ed11745168cd67ee403ff098e reported cstate change: config changed from index 13 to 15, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New cstate: current_term: 3 leader_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:29.331666 29873 raft_consensus.cc:2932] T 14037ba64754436ba114928a3e0dcbe0 P 34a8610adef8401085b62d8fb1fbc8fe [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "0773aa6e61f442bb86d966ae9e97cbc6" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 39123 } attrs { promote: false } } }
I20240502 01:58:29.334494 28391 tablet_service.cc:1514] Processing DeleteTablet for tablet 14037ba64754436ba114928a3e0dcbe0 with delete_type TABLET_DATA_TOMBSTONED (TS 8f4edf52bb2d4624ae596b032e50c2a4 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:36436
I20240502 01:58:29.335160 32131 tablet_replica.cc:330] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: stopping tablet replica
I20240502 01:58:29.335302 32131 raft_consensus.cc:2229] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:29.335410 32131 raft_consensus.cc:2258] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:29.335920 32131 ts_tablet_manager.cc:1875] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:29.337706 32131 ts_tablet_manager.cc:1888] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20240502 01:58:29.337772 32131 log.cc:1196] T 14037ba64754436ba114928a3e0dcbe0 P 8f4edf52bb2d4624ae596b032e50c2a4: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/14037ba64754436ba114928a3e0dcbe0
I20240502 01:58:29.338268 28122 catalog_manager.cc:4830] TS 8f4edf52bb2d4624ae596b032e50c2a4 (127.27.111.194:35369): tablet 14037ba64754436ba114928a3e0dcbe0 (table rebalance_test_table_4 [id=6a3a04e2234e47ff9c73333582341e02]) successfully deleted
W20240502 01:58:29.629842 28607 consensus_peers.cc:485] T ba657ea33b9f445e88d3f1ea97c843a2 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:58:29.683075 29873 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe [term 3 FOLLOWER]: Refusing update from remote peer a1a0ca3ab1ca4872b2bb4395068b7033: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 01:58:29.683526 31862 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:58:29.709522 30004 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 3 FOLLOWER]: Refusing update from remote peer a1a0ca3ab1ca4872b2bb4395068b7033: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 01:58:29.710011 31862 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:58:29.713012 28674 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 3 FOLLOWER]: Refusing update from remote peer a1a0ca3ab1ca4872b2bb4395068b7033: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 01:58:29.714062 32125 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:58:29.715497 29742 consensus_queue.cc:237] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } }
I20240502 01:58:29.717217 29873 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe [term 3 FOLLOWER]: Refusing update from remote peer a1a0ca3ab1ca4872b2bb4395068b7033: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 01:58:29.717326 30004 raft_consensus.cc:1261] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 3 FOLLOWER]: Refusing update from remote peer a1a0ca3ab1ca4872b2bb4395068b7033: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 01:58:29.717767 32156 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:58:29.717912 32125 consensus_queue.cc:1035] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:58:29.719501 32155 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 8cb8626af432419c922ceefff85f69f2 (127.27.111.196) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:29.719862 30004 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P dd5f234ed11745168cd67ee403ff098e [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 8cb8626af432419c922ceefff85f69f2 (127.27.111.196) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:29.720105 29873 raft_consensus.cc:2932] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 34a8610adef8401085b62d8fb1fbc8fe [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 8cb8626af432419c922ceefff85f69f2 (127.27.111.196) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } } }
I20240502 01:58:29.721299 28125 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:58:29.721869 28130 catalog_manager.cc:5484] T 5e87ab8d205e43098dd84ae5f4eeeb6c P a1a0ca3ab1ca4872b2bb4395068b7033 reported cstate change: config changed from index 12 to 14, VOTER 8cb8626af432419c922ceefff85f69f2 (127.27.111.196) evicted. New cstate: current_term: 3 leader_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "dd5f234ed11745168cd67ee403ff098e" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 42203 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1a0ca3ab1ca4872b2bb4395068b7033" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 40085 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:58:29.723521 28476 consensus_peers.cc:485] T cb5e784833984a5a95c1340c96d15994 P ef3bc5c942d04bd29540a80415308353 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:58:29.725020 28654 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e87ab8d205e43098dd84ae5f4eeeb6c with delete_type TABLET_DATA_TOMBSTONED (TS 8cb8626af432419c922ceefff85f69f2 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:49092
I20240502 01:58:29.725346 32158 tablet_replica.cc:330] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2: stopping tablet replica
I20240502 01:58:29.725489 32158 raft_consensus.cc:2229] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:29.725595 32158 raft_consensus.cc:2258] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:29.726186 32158 ts_tablet_manager.cc:1875] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:29.727895 32158 ts_tablet_manager.cc:1888] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240502 01:58:29.727959 32158 log.cc:1196] T 5e87ab8d205e43098dd84ae5f4eeeb6c P 8cb8626af432419c922ceefff85f69f2: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/5e87ab8d205e43098dd84ae5f4eeeb6c
I20240502 01:58:29.728459 28125 catalog_manager.cc:4830] TS 8cb8626af432419c922ceefff85f69f2 (127.27.111.196:40797): tablet 5e87ab8d205e43098dd84ae5f4eeeb6c (table rebalance_test_table_1 [id=eae4e7ef163a4d2788386ab9c9fd7841]) successfully deleted
W20240502 01:58:29.744987 28345 consensus_peers.cc:485] T 2efc1c7671084686ab57f6943aa60134 P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:58:29.876322 28607 consensus_peers.cc:485] T bb71470fc7c54c0490a156fd4256ff48 P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:58:29.881048 30115 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:29.887073 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:29.888813 29722 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:29.888975 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:29.894551 28523 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:29.904081 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:29.912320 28391 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:30.003659 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:30.004947 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:30.007556 30115 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:30.011579 28391 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:30.011932 28523 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:30.015450 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:30.029989 29722 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:30.064662 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.193:34973
--local_ip_for_outbound_sockets=127.27.111.193
--tserver_master_addrs=127.27.111.254:42667
--webserver_port=35723
--webserver_interface=127.27.111.193
--builtin_ntp_servers=127.27.111.212:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 01:58:30.146163 28345 consensus_peers.cc:485] T a14a4e6aa65240a4a1ea29bd564521ab P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:58:30.179952 32234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:30.180195 32234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:30.180274 32234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:30.183393 32234 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:58:30.183470 32234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:30.183581 32234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.193
I20240502 01:58:30.187333 32234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.193:34973
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.111.193
--webserver_port=35723
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.27.111.254:42667
--never_fsync=true
--heap_profile_path=/tmp/kudu.32234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.193
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:30.188309 32234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:30.189538 32234 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:30.199082 32239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:30.199889 32242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:30.200484 32240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:30.201210 32234 server_base.cc:1034] running on GCE node
I20240502 01:58:30.201619 32234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:30.202229 32234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:30.203368 32234 hybrid_clock.cc:648] HybridClock initialized: now 1714615110203355 us; error 59 us; skew 500 ppm
I20240502 01:58:30.205209 32234 webserver.cc:460] Webserver started at http://127.27.111.193:35723/ using document root <none> and password file <none>
I20240502 01:58:30.205806 32234 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:30.205878 32234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:30.209264 32234 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:30.212126 32248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:30.213189 32234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:30.213290 32234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a962820b582e4e158a2c765c544b8e52"
format_stamp: "Formatted at 2024-05-02 01:57:56 on dist-test-slave-0pw1"
I20240502 01:58:30.213707 32234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240502 01:58:30.215799 28345 consensus_peers.cc:485] T 1c4edc183396406f81e997e8bf6f53fb P 8f4edf52bb2d4624ae596b032e50c2a4 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.193:34973: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:58:30.230515 32234 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:30.231159 32234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:30.231313 32234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:30.231873 32234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:30.233114 32255 ts_tablet_manager.cc:516] Loading tablet metadata (0/12 complete)
I20240502 01:58:30.243458 32234 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240502 01:58:30.243543 32234 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.011s user 0.001s sys 0.000s
I20240502 01:58:30.243623 32234 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240502 01:58:30.245924 32255 tablet_bootstrap.cc:492] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:30.249898 32234 ts_tablet_manager.cc:590] Registered 12 tablets
I20240502 01:58:30.249962 32234 ts_tablet_manager.cc:569] Time spent register tablets: real 0.006s user 0.003s sys 0.000s
I20240502 01:58:30.260304 32234 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.193:34973
I20240502 01:58:30.260345 32362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.193:34973 every 8 connection(s)
I20240502 01:58:30.261734 32234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:58:30.261957 32255 log.cc:824] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:30.271260 32363 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42667
I20240502 01:58:30.271301 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 32234
I20240502 01:58:30.271498 32363 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:30.272007 32363 heartbeater.cc:511] Master 127.27.111.254:42667 requested a full tablet report, sending...
I20240502 01:58:30.274588 28129 ts_manager.cc:194] Re-registered known tserver with Master: a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973)
I20240502 01:58:30.276733 28129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.193:51589
I20240502 01:58:30.292217 32255 tablet_bootstrap.cc:492] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:30.292703 32255 tablet_bootstrap.cc:492] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52: Bootstrap complete.
I20240502 01:58:30.294080 32255 ts_tablet_manager.cc:1367] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.049s user 0.040s sys 0.004s
I20240502 01:58:30.296618 32255 raft_consensus.cc:348] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:30.297034 32255 raft_consensus.cc:729] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.297487 32255 consensus_queue.cc:260] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:30.298063 32363 heartbeater.cc:503] Master 127.27.111.254:42667 was elected leader, sending a full tablet report...
I20240502 01:58:30.298267 32255 ts_tablet_manager.cc:1398] T bb71470fc7c54c0490a156fd4256ff48 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20240502 01:58:30.298529 32255 tablet_bootstrap.cc:492] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
W20240502 01:58:30.313733 28607 consensus_peers.cc:485] T f2518b866b1743e28cf63dfe56dbdf3e P 8cb8626af432419c922ceefff85f69f2 -> Peer a962820b582e4e158a2c765c544b8e52 (127.27.111.193:34973): Couldn't send request to peer a962820b582e4e158a2c765c544b8e52. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
I20240502 01:58:30.328311 32255 tablet_bootstrap.cc:492] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:30.328722 32255 tablet_bootstrap.cc:492] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52: Bootstrap complete.
I20240502 01:58:30.329869 32255 ts_tablet_manager.cc:1367] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.008s
I20240502 01:58:30.330507 32255 raft_consensus.cc:348] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:30.330667 32255 raft_consensus.cc:729] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.330816 32255 consensus_queue.cc:260] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:30.331073 32255 ts_tablet_manager.cc:1398] T f2518b866b1743e28cf63dfe56dbdf3e P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.331332 32255 tablet_bootstrap.cc:492] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:30.356103 32255 tablet_bootstrap.cc:492] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:30.356456 32255 tablet_bootstrap.cc:492] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52: Bootstrap complete.
I20240502 01:58:30.357415 32255 ts_tablet_manager.cc:1367] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.007s
I20240502 01:58:30.357915 32255 raft_consensus.cc:348] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:30.358038 32255 raft_consensus.cc:729] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.358222 32255 consensus_queue.cc:260] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:30.358556 32255 ts_tablet_manager.cc:1398] T cb5e784833984a5a95c1340c96d15994 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240502 01:58:30.358759 32255 tablet_bootstrap.cc:492] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:30.382867 32255 tablet_bootstrap.cc:492] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:30.383306 32255 tablet_bootstrap.cc:492] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52: Bootstrap complete.
I20240502 01:58:30.384289 32255 ts_tablet_manager.cc:1367] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.007s
I20240502 01:58:30.384742 32255 raft_consensus.cc:348] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:30.384848 32255 raft_consensus.cc:729] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.384997 32255 consensus_queue.cc:260] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:30.385272 32255 ts_tablet_manager.cc:1398] T 4f816e20fed94a0587d9c7cfac7071cc P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:30.385465 32255 tablet_bootstrap.cc:492] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:30.413777 32255 tablet_bootstrap.cc:492] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:30.414177 32255 tablet_bootstrap.cc:492] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52: Bootstrap complete.
I20240502 01:58:30.415359 32255 ts_tablet_manager.cc:1367] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.006s
I20240502 01:58:30.415814 32255 raft_consensus.cc:348] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:30.415949 32255 raft_consensus.cc:729] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.416103 32255 consensus_queue.cc:260] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } }
I20240502 01:58:30.416385 32255 ts_tablet_manager.cc:1398] T ba657ea33b9f445e88d3f1ea97c843a2 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:30.416538 32255 tablet_bootstrap.cc:492] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:30.441622 32255 tablet_bootstrap.cc:492] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:30.442031 32255 tablet_bootstrap.cc:492] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52: Bootstrap complete.
I20240502 01:58:30.443193 32255 ts_tablet_manager.cc:1367] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.007s
I20240502 01:58:30.443806 32255 raft_consensus.cc:348] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:30.443964 32255 raft_consensus.cc:729] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.444162 32255 consensus_queue.cc:260] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } }
I20240502 01:58:30.444591 32255 ts_tablet_manager.cc:1398] T a14a4e6aa65240a4a1ea29bd564521ab P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.444830 32255 tablet_bootstrap.cc:492] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:30.478107 32255 tablet_bootstrap.cc:492] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:30.478554 32255 tablet_bootstrap.cc:492] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52: Bootstrap complete.
I20240502 01:58:30.479794 32255 ts_tablet_manager.cc:1367] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.005s
I20240502 01:58:30.480326 32255 raft_consensus.cc:348] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:30.480439 32255 raft_consensus.cc:729] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.480612 32255 consensus_queue.cc:260] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:30.481014 32255 ts_tablet_manager.cc:1398] T 1c4edc183396406f81e997e8bf6f53fb P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:30.481343 32255 tablet_bootstrap.cc:492] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:30.496886 30115 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:30.499609 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:30.508383 32297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:30.508738 28391 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:30.517343 29722 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:30.518220 32255 tablet_bootstrap.cc:492] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:30.518736 32255 tablet_bootstrap.cc:492] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52: Bootstrap complete.
I20240502 01:58:30.520000 32255 ts_tablet_manager.cc:1367] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.039s user 0.019s sys 0.016s
I20240502 01:58:30.520495 32255 raft_consensus.cc:348] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:30.520684 32255 raft_consensus.cc:729] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.520893 32255 consensus_queue.cc:260] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:30.521751 32255 ts_tablet_manager.cc:1398] T fa11ebedbf5c4038a5388d9440688011 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:30.521957 32255 tablet_bootstrap.cc:492] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:30.522708 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:30.530448 28523 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:30.556161 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:30.561259 32255 tablet_bootstrap.cc:492] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:30.561671 32255 tablet_bootstrap.cc:492] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52: Bootstrap complete.
I20240502 01:58:30.562885 32255 ts_tablet_manager.cc:1367] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.041s user 0.031s sys 0.002s
I20240502 01:58:30.563370 32255 raft_consensus.cc:348] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:30.563515 32255 raft_consensus.cc:729] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.563688 32255 consensus_queue.cc:260] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:30.563994 32255 ts_tablet_manager.cc:1398] T 2efc1c7671084686ab57f6943aa60134 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:30.564198 32255 tablet_bootstrap.cc:492] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:30.595219 32255 tablet_bootstrap.cc:492] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:30.595650 32255 tablet_bootstrap.cc:492] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52: Bootstrap complete.
I20240502 01:58:30.596722 32255 ts_tablet_manager.cc:1367] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.033s user 0.031s sys 0.000s
I20240502 01:58:30.597160 32255 raft_consensus.cc:348] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:30.597277 32255 raft_consensus.cc:729] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.597396 32255 consensus_queue.cc:260] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } }
I20240502 01:58:30.597683 32255 ts_tablet_manager.cc:1398] T cd3a11e1ac624f3fbc8cacf351b8afa5 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:30.597826 32255 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:30.625574 32255 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:30.625953 32255 tablet_bootstrap.cc:492] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52: Bootstrap complete.
I20240502 01:58:30.627164 32255 ts_tablet_manager.cc:1367] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.007s
I20240502 01:58:30.627701 32255 raft_consensus.cc:348] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:30.627872 32255 raft_consensus.cc:729] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.628013 32255 consensus_queue.cc:260] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } attrs { replace: true } } peers { permanent_uuid: "8f4edf52bb2d4624ae596b032e50c2a4" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 35369 } } peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "34a8610adef8401085b62d8fb1fbc8fe" member_type: NON_VOTER last_known_addr { host: "127.27.111.198" port: 43065 } attrs { promote: true } }
I20240502 01:58:30.628319 32255 ts_tablet_manager.cc:1398] T 14037ba64754436ba114928a3e0dcbe0 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:30.628499 32255 tablet_bootstrap.cc:492] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52: Bootstrap starting.
I20240502 01:58:30.663489 29853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:30.664610 32255 tablet_bootstrap.cc:492] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:30.665022 32255 tablet_bootstrap.cc:492] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52: Bootstrap complete.
I20240502 01:58:30.666249 32255 ts_tablet_manager.cc:1367] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52: Time spent bootstrapping tablet: real 0.038s user 0.032s sys 0.000s
I20240502 01:58:30.666779 32255 raft_consensus.cc:348] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:30.666911 32255 raft_consensus.cc:729] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a962820b582e4e158a2c765c544b8e52, State: Initialized, Role: FOLLOWER
I20240502 01:58:30.667054 32255 consensus_queue.cc:260] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cb8626af432419c922ceefff85f69f2" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 40797 } } peers { permanent_uuid: "a962820b582e4e158a2c765c544b8e52" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 34973 } } peers { permanent_uuid: "ef3bc5c942d04bd29540a80415308353" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 46695 } }
I20240502 01:58:30.667340 32255 ts_tablet_manager.cc:1398] T 803991b0b27f470dbfe2397baa997621 P a962820b582e4e158a2c765c544b8e52: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 01:58:30.669852 29983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:30.677172 28391 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:30.677990 28654 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:30.681053 32297 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:30.690158 30115 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:30.694221 29722 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:30.700709 28523 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:30.745550 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 32234
I20240502 01:58:30.752416 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 28330
I20240502 01:58:30.759845 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 28461
I20240502 01:58:30.768460 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 28592
I20240502 01:58:30.775430 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 29660
I20240502 01:58:30.781942 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 29791
I20240502 01:58:30.788038 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 29922
I20240502 01:58:30.793656 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 30053
I20240502 01:58:30.799472 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 28107
2024-05-02T01:58:30Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (35068 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (35068 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240502 01:58:30.833853 28095 test_util.cc:268] Using random seed: -258238792
I20240502 01:58:30.834661 28095 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:58:30.834702 28095 ts_itest-base.cc:113] --------------
I20240502 01:58:30.834713 28095 ts_itest-base.cc:114] 5 tablet servers
I20240502 01:58:30.834723 28095 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:58:30.834734 28095 ts_itest-base.cc:116] --------------
2024-05-02T01:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:58:30Z Disabled control of system clock
I20240502 01:58:30.842442 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.111.254:35241
--webserver_interface=127.27.111.254
--webserver_port=0
--builtin_ntp_servers=127.27.111.212:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.111.254:35241
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 01:58:30.948177 32466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:30.948451 32466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:30.948525 32466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:30.951619 32466 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:30.951695 32466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:58:30.951735 32466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:30.951776 32466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:58:30.951817 32466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:58:30.955468 32466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.111.254:35241
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.111.254:35241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.111.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.32466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:30.956429 32466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:30.957558 32466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:30.965409 32474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:30.965920 32472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:30.966537 32471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:30.967924 32466 server_base.cc:1034] running on GCE node
I20240502 01:58:30.968432 32466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:30.969357 32466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:30.970508 32466 hybrid_clock.cc:648] HybridClock initialized: now 1714615110970500 us; error 25 us; skew 500 ppm
I20240502 01:58:30.972577 32466 webserver.cc:460] Webserver started at http://127.27.111.254:44637/ using document root <none> and password file <none>
I20240502 01:58:30.973160 32466 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:30.973223 32466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:30.973402 32466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:30.975236 32466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8efa3ac9a929478d980e9039b58c3850"
format_stamp: "Formatted at 2024-05-02 01:58:30 on dist-test-slave-0pw1"
I20240502 01:58:30.975713 32466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8efa3ac9a929478d980e9039b58c3850"
format_stamp: "Formatted at 2024-05-02 01:58:30 on dist-test-slave-0pw1"
I20240502 01:58:30.979220 32466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:58:30.981308 32480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:30.982275 32466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:30.982474 32466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8efa3ac9a929478d980e9039b58c3850"
format_stamp: "Formatted at 2024-05-02 01:58:30 on dist-test-slave-0pw1"
I20240502 01:58:30.982578 32466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:30.999608 32466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:31.000286 32466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:31.000458 32466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:31.006906 32466 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.254:35241
I20240502 01:58:31.006928 32532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.254:35241 every 8 connection(s)
I20240502 01:58:31.007884 32466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:58:31.010424 32533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:31.015370 32533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850: Bootstrap starting.
I20240502 01:58:31.017261 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 32466
I20240502 01:58:31.017371 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:58:31.017596 32533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:31.018311 32533 log.cc:824] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:31.020015 32533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850: No bootstrap required, opened a new log
I20240502 01:58:31.022886 32533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efa3ac9a929478d980e9039b58c3850" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 35241 } }
I20240502 01:58:31.023072 32533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:31.023124 32533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8efa3ac9a929478d980e9039b58c3850, State: Initialized, Role: FOLLOWER
I20240502 01:58:31.023535 32533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efa3ac9a929478d980e9039b58c3850" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 35241 } }
I20240502 01:58:31.023649 32533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:31.023700 32533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:31.023810 32533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:31.024602 32533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efa3ac9a929478d980e9039b58c3850" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 35241 } }
I20240502 01:58:31.024941 32533 leader_election.cc:302] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8efa3ac9a929478d980e9039b58c3850; no voters:
I20240502 01:58:31.025314 32533 leader_election.cc:288] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:31.025426 32538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:31.025609 32538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [term 1 LEADER]: Becoming Leader. State: Replica: 8efa3ac9a929478d980e9039b58c3850, State: Running, Role: LEADER
I20240502 01:58:31.025882 32538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efa3ac9a929478d980e9039b58c3850" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 35241 } }
I20240502 01:58:31.026530 32533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:58:31.026983 32540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8efa3ac9a929478d980e9039b58c3850. Latest consensus state: current_term: 1 leader_uuid: "8efa3ac9a929478d980e9039b58c3850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efa3ac9a929478d980e9039b58c3850" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 35241 } } }
I20240502 01:58:31.027114 32540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:31.027506 32542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:58:31.027670 32539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8efa3ac9a929478d980e9039b58c3850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efa3ac9a929478d980e9039b58c3850" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 35241 } } }
I20240502 01:58:31.027794 32539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:31.030576 32542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:58:31.036176 32542 catalog_manager.cc:1347] Generated new cluster ID: 305563b4f57b474f8e74b8d4904737ad
I20240502 01:58:31.036237 32542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:58:31.050026 32542 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:58:31.051038 32542 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:58:31.057021 32542 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8efa3ac9a929478d980e9039b58c3850: Generated new TSK 0
I20240502 01:58:31.057662 32542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:58:31.061023 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.193:0
--local_ip_for_outbound_sockets=127.27.111.193
--webserver_interface=127.27.111.193
--webserver_port=0
--tserver_master_addrs=127.27.111.254:35241
--builtin_ntp_servers=127.27.111.212:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:58:31.178894 32557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:31.179139 32557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:31.179186 32557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:31.182263 32557 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:31.182336 32557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:31.182443 32557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.193
I20240502 01:58:31.186069 32557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.111.193
--webserver_port=0
--tserver_master_addrs=127.27.111.254:35241
--never_fsync=true
--heap_profile_path=/tmp/kudu.32557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.193
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:31.187067 32557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:31.188230 32557 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:31.197363 32563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:31.197952 32565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:31.198601 32562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:31.199565 32557 server_base.cc:1034] running on GCE node
I20240502 01:58:31.200004 32557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:31.200575 32557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:31.201748 32557 hybrid_clock.cc:648] HybridClock initialized: now 1714615111201729 us; error 38 us; skew 500 ppm
I20240502 01:58:31.203752 32557 webserver.cc:460] Webserver started at http://127.27.111.193:46079/ using document root <none> and password file <none>
I20240502 01:58:31.204283 32557 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:31.204336 32557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:31.204492 32557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:31.206122 32557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "47fe3b51d11b43fbb68769d924f4ba26"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.206602 32557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "47fe3b51d11b43fbb68769d924f4ba26"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.209839 32557 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:31.211937 32571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.212867 32557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 01:58:31.212958 32557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "47fe3b51d11b43fbb68769d924f4ba26"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.213027 32557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:31.234156 32557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:31.234826 32557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:31.234966 32557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:31.235445 32557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:31.236294 32557 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:31.236340 32557 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.236387 32557 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:31.236409 32557 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.244518 32557 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.193:42259
I20240502 01:58:31.244560 32684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.193:42259 every 8 connection(s)
I20240502 01:58:31.245438 32557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:58:31.250430 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 32557
I20240502 01:58:31.250558 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:58:31.252921 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.194:0
--local_ip_for_outbound_sockets=127.27.111.194
--webserver_interface=127.27.111.194
--webserver_port=0
--tserver_master_addrs=127.27.111.254:35241
--builtin_ntp_servers=127.27.111.212:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:58:31.254218 32685 heartbeater.cc:347] Connected to a master server at 127.27.111.254:35241
I20240502 01:58:31.254536 32685 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:31.255126 32685 heartbeater.cc:511] Master 127.27.111.254:35241 requested a full tablet report, sending...
I20240502 01:58:31.256492 32497 ts_manager.cc:194] Registered new tserver with Master: 47fe3b51d11b43fbb68769d924f4ba26 (127.27.111.193:42259)
I20240502 01:58:31.257671 32497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.193:46699
W20240502 01:58:31.367038 32688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:31.367250 32688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:31.367295 32688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:31.370332 32688 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:31.370409 32688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:31.370488 32688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.194
I20240502 01:58:31.374064 32688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.111.194
--webserver_port=0
--tserver_master_addrs=127.27.111.254:35241
--never_fsync=true
--heap_profile_path=/tmp/kudu.32688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.194
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:31.374950 32688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:31.376062 32688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:31.384888 32696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:31.385604 32693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:31.386466 32694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:31.386916 32688 server_base.cc:1034] running on GCE node
I20240502 01:58:31.387295 32688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:31.387820 32688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:31.388989 32688 hybrid_clock.cc:648] HybridClock initialized: now 1714615111388978 us; error 26 us; skew 500 ppm
I20240502 01:58:31.390971 32688 webserver.cc:460] Webserver started at http://127.27.111.194:35487/ using document root <none> and password file <none>
I20240502 01:58:31.391523 32688 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:31.391587 32688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:31.391762 32688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:31.393427 32688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "243a928cd9964e3bad0d41f297541754"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.393931 32688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "243a928cd9964e3bad0d41f297541754"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.397215 32688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:58:31.399456 32702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.400441 32688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:31.400552 32688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "243a928cd9964e3bad0d41f297541754"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.400636 32688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:31.416504 32688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:31.417086 32688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:31.417244 32688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:31.417812 32688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:31.418757 32688 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:31.418828 32688 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.418884 32688 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:31.418918 32688 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.427454 32688 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.194:43607
I20240502 01:58:31.427490 347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.194:43607 every 8 connection(s)
I20240502 01:58:31.428433 32688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:58:31.429530 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 32688
I20240502 01:58:31.429625 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:58:31.431927 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.195:0
--local_ip_for_outbound_sockets=127.27.111.195
--webserver_interface=127.27.111.195
--webserver_port=0
--tserver_master_addrs=127.27.111.254:35241
--builtin_ntp_servers=127.27.111.212:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:58:31.438294 348 heartbeater.cc:347] Connected to a master server at 127.27.111.254:35241
I20240502 01:58:31.438540 348 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:31.439004 348 heartbeater.cc:511] Master 127.27.111.254:35241 requested a full tablet report, sending...
I20240502 01:58:31.440091 32497 ts_manager.cc:194] Registered new tserver with Master: 243a928cd9964e3bad0d41f297541754 (127.27.111.194:43607)
I20240502 01:58:31.440927 32497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.194:51923
W20240502 01:58:31.545496 351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:31.545763 351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:31.545810 351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:31.548920 351 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:31.549000 351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:31.549070 351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.195
I20240502 01:58:31.552767 351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.111.195
--webserver_port=0
--tserver_master_addrs=127.27.111.254:35241
--never_fsync=true
--heap_profile_path=/tmp/kudu.351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.195
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:31.553714 351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:31.554948 351 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:31.563777 357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:31.564198 356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:31.564769 359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:31.565721 351 server_base.cc:1034] running on GCE node
I20240502 01:58:31.566145 351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:31.566753 351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:31.567950 351 hybrid_clock.cc:648] HybridClock initialized: now 1714615111567938 us; error 31 us; skew 500 ppm
I20240502 01:58:31.569914 351 webserver.cc:460] Webserver started at http://127.27.111.195:41323/ using document root <none> and password file <none>
I20240502 01:58:31.570515 351 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:31.570576 351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:31.570748 351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:31.572435 351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7b98171c8198455aa773391897ea2b70"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.572911 351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7b98171c8198455aa773391897ea2b70"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.576366 351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 01:58:31.578610 365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.579725 351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:31.579841 351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7b98171c8198455aa773391897ea2b70"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.579937 351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:31.592197 351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:31.592820 351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:31.593003 351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:31.593514 351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:31.594461 351 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:31.594511 351 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.594550 351 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:31.594571 351 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.602773 351 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.195:37349
I20240502 01:58:31.602819 478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.195:37349 every 8 connection(s)
I20240502 01:58:31.605043 351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:58:31.608752 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 351
I20240502 01:58:31.608870 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:58:31.611302 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.196:0
--local_ip_for_outbound_sockets=127.27.111.196
--webserver_interface=127.27.111.196
--webserver_port=0
--tserver_master_addrs=127.27.111.254:35241
--builtin_ntp_servers=127.27.111.212:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:58:31.616734 479 heartbeater.cc:347] Connected to a master server at 127.27.111.254:35241
I20240502 01:58:31.617012 479 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:31.617480 479 heartbeater.cc:511] Master 127.27.111.254:35241 requested a full tablet report, sending...
I20240502 01:58:31.618503 32497 ts_manager.cc:194] Registered new tserver with Master: 7b98171c8198455aa773391897ea2b70 (127.27.111.195:37349)
I20240502 01:58:31.619061 32497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.195:46967
W20240502 01:58:31.722644 482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:31.722929 482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:31.723016 482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:31.726171 482 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:31.726258 482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:31.726374 482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.196
I20240502 01:58:31.730520 482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.111.196
--webserver_port=0
--tserver_master_addrs=127.27.111.254:35241
--never_fsync=true
--heap_profile_path=/tmp/kudu.482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.196
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:31.731504 482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:31.732707 482 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:31.742029 487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:31.742789 488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:31.743394 490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:31.744025 482 server_base.cc:1034] running on GCE node
I20240502 01:58:31.744450 482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:31.745059 482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:31.746215 482 hybrid_clock.cc:648] HybridClock initialized: now 1714615111746204 us; error 26 us; skew 500 ppm
I20240502 01:58:31.748329 482 webserver.cc:460] Webserver started at http://127.27.111.196:40067/ using document root <none> and password file <none>
I20240502 01:58:31.748919 482 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:31.748980 482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:31.749172 482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:31.750947 482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d206618059b843e8a77c4de818efdbc1"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.751435 482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d206618059b843e8a77c4de818efdbc1"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.755052 482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:58:31.757409 496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.758491 482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:58:31.758603 482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d206618059b843e8a77c4de818efdbc1"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.758699 482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:31.769953 482 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:31.770653 482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:31.770814 482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:31.771344 482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:31.772230 482 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:31.772279 482 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.772334 482 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:31.772365 482 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.781049 482 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.196:36965
I20240502 01:58:31.781091 609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.196:36965 every 8 connection(s)
I20240502 01:58:31.782043 482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:58:31.787866 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 482
I20240502 01:58:31.787992 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:58:31.790282 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.197:0
--local_ip_for_outbound_sockets=127.27.111.197
--webserver_interface=127.27.111.197
--webserver_port=0
--tserver_master_addrs=127.27.111.254:35241
--builtin_ntp_servers=127.27.111.212:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:58:31.791780 610 heartbeater.cc:347] Connected to a master server at 127.27.111.254:35241
I20240502 01:58:31.792085 610 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:31.792716 610 heartbeater.cc:511] Master 127.27.111.254:35241 requested a full tablet report, sending...
I20240502 01:58:31.793933 32497 ts_manager.cc:194] Registered new tserver with Master: d206618059b843e8a77c4de818efdbc1 (127.27.111.196:36965)
I20240502 01:58:31.794610 32497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.196:52557
W20240502 01:58:31.901520 613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:31.901785 613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:31.901842 613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:31.905162 613 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:31.905241 613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:31.905344 613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.197
I20240502 01:58:31.909019 613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.111.197
--webserver_port=0
--tserver_master_addrs=127.27.111.254:35241
--never_fsync=true
--heap_profile_path=/tmp/kudu.613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.197
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:31.909976 613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:31.911211 613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:31.920135 618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:31.920876 621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:31.921638 619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:31.922276 613 server_base.cc:1034] running on GCE node
I20240502 01:58:31.922799 613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:31.923417 613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:31.924597 613 hybrid_clock.cc:648] HybridClock initialized: now 1714615111924581 us; error 29 us; skew 500 ppm
I20240502 01:58:31.926787 613 webserver.cc:460] Webserver started at http://127.27.111.197:46149/ using document root <none> and password file <none>
I20240502 01:58:31.927412 613 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:31.927486 613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:31.927681 613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:31.929503 613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "432f171089fb4f67a46a5db2f9c2c610"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.929981 613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "432f171089fb4f67a46a5db2f9c2c610"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.933566 613 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 01:58:31.936023 627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.937179 613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:31.937297 613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "432f171089fb4f67a46a5db2f9c2c610"
format_stamp: "Formatted at 2024-05-02 01:58:31 on dist-test-slave-0pw1"
I20240502 01:58:31.937422 613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:31.953720 613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:31.954367 613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:31.954557 613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:31.955101 613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:31.956040 613 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:31.956089 613 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.956136 613 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:31.956169 613 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:31.965457 613 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.197:46685
I20240502 01:58:31.965485 740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.197:46685 every 8 connection(s)
I20240502 01:58:31.966533 613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:58:31.967006 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 613
I20240502 01:58:31.967145 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:58:31.975857 741 heartbeater.cc:347] Connected to a master server at 127.27.111.254:35241
I20240502 01:58:31.976066 741 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:31.976487 741 heartbeater.cc:511] Master 127.27.111.254:35241 requested a full tablet report, sending...
I20240502 01:58:31.977437 32497 ts_manager.cc:194] Registered new tserver with Master: 432f171089fb4f67a46a5db2f9c2c610 (127.27.111.197:46685)
I20240502 01:58:31.977994 32497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.197:41199
I20240502 01:58:31.981668 28095 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:58:31.990633 32497 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46108:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 01:58:32.008873 675 tablet_service.cc:1467] Processing CreateTablet for tablet 5682d584bf8049ba95316340fada03bc (DEFAULT_TABLE table=TestTable [id=f68cb3cff34f426595f2974c3961d6a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:32.008872 544 tablet_service.cc:1467] Processing CreateTablet for tablet 5682d584bf8049ba95316340fada03bc (DEFAULT_TABLE table=TestTable [id=f68cb3cff34f426595f2974c3961d6a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:32.008872 32613 tablet_service.cc:1467] Processing CreateTablet for tablet 5682d584bf8049ba95316340fada03bc (DEFAULT_TABLE table=TestTable [id=f68cb3cff34f426595f2974c3961d6a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:32.009863 544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5682d584bf8049ba95316340fada03bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:32.009862 32613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5682d584bf8049ba95316340fada03bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:32.009868 675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5682d584bf8049ba95316340fada03bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:32.017175 759 tablet_bootstrap.cc:492] T 5682d584bf8049ba95316340fada03bc P d206618059b843e8a77c4de818efdbc1: Bootstrap starting.
I20240502 01:58:32.017269 760 tablet_bootstrap.cc:492] T 5682d584bf8049ba95316340fada03bc P 432f171089fb4f67a46a5db2f9c2c610: Bootstrap starting.
I20240502 01:58:32.018769 758 tablet_bootstrap.cc:492] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26: Bootstrap starting.
I20240502 01:58:32.019570 759 tablet_bootstrap.cc:654] T 5682d584bf8049ba95316340fada03bc P d206618059b843e8a77c4de818efdbc1: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:32.020176 760 tablet_bootstrap.cc:654] T 5682d584bf8049ba95316340fada03bc P 432f171089fb4f67a46a5db2f9c2c610: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:32.020470 759 log.cc:824] T 5682d584bf8049ba95316340fada03bc P d206618059b843e8a77c4de818efdbc1: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:32.020934 760 log.cc:824] T 5682d584bf8049ba95316340fada03bc P 432f171089fb4f67a46a5db2f9c2c610: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:32.021062 758 tablet_bootstrap.cc:654] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:32.021750 758 log.cc:824] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:32.022106 759 tablet_bootstrap.cc:492] T 5682d584bf8049ba95316340fada03bc P d206618059b843e8a77c4de818efdbc1: No bootstrap required, opened a new log
I20240502 01:58:32.022295 759 ts_tablet_manager.cc:1367] T 5682d584bf8049ba95316340fada03bc P d206618059b843e8a77c4de818efdbc1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:58:32.023447 758 tablet_bootstrap.cc:492] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26: No bootstrap required, opened a new log
I20240502 01:58:32.023633 758 ts_tablet_manager.cc:1367] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:58:32.024983 760 tablet_bootstrap.cc:492] T 5682d584bf8049ba95316340fada03bc P 432f171089fb4f67a46a5db2f9c2c610: No bootstrap required, opened a new log
I20240502 01:58:32.025154 760 ts_tablet_manager.cc:1367] T 5682d584bf8049ba95316340fada03bc P 432f171089fb4f67a46a5db2f9c2c610: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 01:58:32.025785 759 raft_consensus.cc:348] T 5682d584bf8049ba95316340fada03bc P d206618059b843e8a77c4de818efdbc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d206618059b843e8a77c4de818efdbc1" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36965 } } peers { permanent_uuid: "47fe3b51d11b43fbb68769d924f4ba26" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42259 } } peers { permanent_uuid: "432f171089fb4f67a46a5db2f9c2c610" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 46685 } }
I20240502 01:58:32.026011 759 raft_consensus.cc:374] T 5682d584bf8049ba95316340fada03bc P d206618059b843e8a77c4de818efdbc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:32.026070 759 raft_consensus.cc:729] T 5682d584bf8049ba95316340fada03bc P d206618059b843e8a77c4de818efdbc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d206618059b843e8a77c4de818efdbc1, State: Initialized, Role: FOLLOWER
I20240502 01:58:32.026547 759 consensus_queue.cc:260] T 5682d584bf8049ba95316340fada03bc P d206618059b843e8a77c4de818efdbc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d206618059b843e8a77c4de818efdbc1" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36965 } } peers { permanent_uuid: "47fe3b51d11b43fbb68769d924f4ba26" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42259 } } peers { permanent_uuid: "432f171089fb4f67a46a5db2f9c2c610" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 46685 } }
I20240502 01:58:32.027232 610 heartbeater.cc:503] Master 127.27.111.254:35241 was elected leader, sending a full tablet report...
I20240502 01:58:32.027541 759 ts_tablet_manager.cc:1398] T 5682d584bf8049ba95316340fada03bc P d206618059b843e8a77c4de818efdbc1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 01:58:32.027041 758 raft_consensus.cc:348] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d206618059b843e8a77c4de818efdbc1" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36965 } } peers { permanent_uuid: "47fe3b51d11b43fbb68769d924f4ba26" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42259 } } peers { permanent_uuid: "432f171089fb4f67a46a5db2f9c2c610" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 46685 } }
I20240502 01:58:32.028281 758 raft_consensus.cc:374] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:32.028378 758 raft_consensus.cc:729] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47fe3b51d11b43fbb68769d924f4ba26, State: Initialized, Role: FOLLOWER
I20240502 01:58:32.028803 758 consensus_queue.cc:260] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d206618059b843e8a77c4de818efdbc1" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36965 } } peers { permanent_uuid: "47fe3b51d11b43fbb68769d924f4ba26" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42259 } } peers { permanent_uuid: "432f171089fb4f67a46a5db2f9c2c610" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 46685 } }
I20240502 01:58:32.028797 760 raft_consensus.cc:348] T 5682d584bf8049ba95316340fada03bc P 432f171089fb4f67a46a5db2f9c2c610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d206618059b843e8a77c4de818efdbc1" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36965 } } peers { permanent_uuid: "47fe3b51d11b43fbb68769d924f4ba26" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42259 } } peers { permanent_uuid: "432f171089fb4f67a46a5db2f9c2c610" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 46685 } }
I20240502 01:58:32.028988 760 raft_consensus.cc:374] T 5682d584bf8049ba95316340fada03bc P 432f171089fb4f67a46a5db2f9c2c610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:32.029040 760 raft_consensus.cc:729] T 5682d584bf8049ba95316340fada03bc P 432f171089fb4f67a46a5db2f9c2c610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432f171089fb4f67a46a5db2f9c2c610, State: Initialized, Role: FOLLOWER
I20240502 01:58:32.029532 32685 heartbeater.cc:503] Master 127.27.111.254:35241 was elected leader, sending a full tablet report...
I20240502 01:58:32.029747 758 ts_tablet_manager.cc:1398] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 01:58:32.030460 760 consensus_queue.cc:260] T 5682d584bf8049ba95316340fada03bc P 432f171089fb4f67a46a5db2f9c2c610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d206618059b843e8a77c4de818efdbc1" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36965 } } peers { permanent_uuid: "47fe3b51d11b43fbb68769d924f4ba26" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42259 } } peers { permanent_uuid: "432f171089fb4f67a46a5db2f9c2c610" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 46685 } }
I20240502 01:58:32.031342 760 ts_tablet_manager.cc:1398] T 5682d584bf8049ba95316340fada03bc P 432f171089fb4f67a46a5db2f9c2c610: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:58:32.031736 741 heartbeater.cc:503] Master 127.27.111.254:35241 was elected leader, sending a full tablet report...
W20240502 01:58:32.032889 611 tablet.cc:2369] T 5682d584bf8049ba95316340fada03bc P d206618059b843e8a77c4de818efdbc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:58:32.217448 742 tablet.cc:2369] T 5682d584bf8049ba95316340fada03bc P 432f171089fb4f67a46a5db2f9c2c610: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:32.232813 765 raft_consensus.cc:482] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:32.233016 765 raft_consensus.cc:504] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d206618059b843e8a77c4de818efdbc1" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36965 } } peers { permanent_uuid: "47fe3b51d11b43fbb68769d924f4ba26" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42259 } } peers { permanent_uuid: "432f171089fb4f67a46a5db2f9c2c610" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 46685 } }
I20240502 01:58:32.234244 765 leader_election.cc:288] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d206618059b843e8a77c4de818efdbc1 (127.27.111.196:36965), 432f171089fb4f67a46a5db2f9c2c610 (127.27.111.197:46685)
I20240502 01:58:32.238675 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5682d584bf8049ba95316340fada03bc" candidate_uuid: "47fe3b51d11b43fbb68769d924f4ba26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d206618059b843e8a77c4de818efdbc1" is_pre_election: true
I20240502 01:58:32.238685 695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5682d584bf8049ba95316340fada03bc" candidate_uuid: "47fe3b51d11b43fbb68769d924f4ba26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432f171089fb4f67a46a5db2f9c2c610" is_pre_election: true
I20240502 01:58:32.238960 564 raft_consensus.cc:2454] T 5682d584bf8049ba95316340fada03bc P d206618059b843e8a77c4de818efdbc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47fe3b51d11b43fbb68769d924f4ba26 in term 0.
I20240502 01:58:32.238991 695 raft_consensus.cc:2454] T 5682d584bf8049ba95316340fada03bc P 432f171089fb4f67a46a5db2f9c2c610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47fe3b51d11b43fbb68769d924f4ba26 in term 0.
I20240502 01:58:32.239424 32573 leader_election.cc:302] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47fe3b51d11b43fbb68769d924f4ba26, d206618059b843e8a77c4de818efdbc1; no voters:
I20240502 01:58:32.239674 765 raft_consensus.cc:2781] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:32.239748 765 raft_consensus.cc:482] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:32.239781 765 raft_consensus.cc:3037] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.240718 765 raft_consensus.cc:504] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d206618059b843e8a77c4de818efdbc1" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36965 } } peers { permanent_uuid: "47fe3b51d11b43fbb68769d924f4ba26" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42259 } } peers { permanent_uuid: "432f171089fb4f67a46a5db2f9c2c610" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 46685 } }
I20240502 01:58:32.241107 765 leader_election.cc:288] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [CANDIDATE]: Term 1 election: Requested vote from peers d206618059b843e8a77c4de818efdbc1 (127.27.111.196:36965), 432f171089fb4f67a46a5db2f9c2c610 (127.27.111.197:46685)
I20240502 01:58:32.241420 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5682d584bf8049ba95316340fada03bc" candidate_uuid: "47fe3b51d11b43fbb68769d924f4ba26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d206618059b843e8a77c4de818efdbc1"
I20240502 01:58:32.241457 695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5682d584bf8049ba95316340fada03bc" candidate_uuid: "47fe3b51d11b43fbb68769d924f4ba26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432f171089fb4f67a46a5db2f9c2c610"
I20240502 01:58:32.241545 564 raft_consensus.cc:3037] T 5682d584bf8049ba95316340fada03bc P d206618059b843e8a77c4de818efdbc1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.241566 695 raft_consensus.cc:3037] T 5682d584bf8049ba95316340fada03bc P 432f171089fb4f67a46a5db2f9c2c610 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:32.242540 695 raft_consensus.cc:2454] T 5682d584bf8049ba95316340fada03bc P 432f171089fb4f67a46a5db2f9c2c610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47fe3b51d11b43fbb68769d924f4ba26 in term 1.
I20240502 01:58:32.242558 564 raft_consensus.cc:2454] T 5682d584bf8049ba95316340fada03bc P d206618059b843e8a77c4de818efdbc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47fe3b51d11b43fbb68769d924f4ba26 in term 1.
I20240502 01:58:32.242830 32573 leader_election.cc:302] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47fe3b51d11b43fbb68769d924f4ba26, d206618059b843e8a77c4de818efdbc1; no voters:
I20240502 01:58:32.243048 765 raft_consensus.cc:2781] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:32.243239 765 raft_consensus.cc:686] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [term 1 LEADER]: Becoming Leader. State: Replica: 47fe3b51d11b43fbb68769d924f4ba26, State: Running, Role: LEADER
I20240502 01:58:32.243537 765 consensus_queue.cc:237] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d206618059b843e8a77c4de818efdbc1" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36965 } } peers { permanent_uuid: "47fe3b51d11b43fbb68769d924f4ba26" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42259 } } peers { permanent_uuid: "432f171089fb4f67a46a5db2f9c2c610" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 46685 } }
I20240502 01:58:32.246454 32497 catalog_manager.cc:5484] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47fe3b51d11b43fbb68769d924f4ba26 (127.27.111.193). New cstate: current_term: 1 leader_uuid: "47fe3b51d11b43fbb68769d924f4ba26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d206618059b843e8a77c4de818efdbc1" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36965 } } peers { permanent_uuid: "47fe3b51d11b43fbb68769d924f4ba26" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42259 } } peers { permanent_uuid: "432f171089fb4f67a46a5db2f9c2c610" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 46685 } } }
W20240502 01:58:32.246846 32686 tablet.cc:2369] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:32.297919 28095 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:58:32.299674 28095 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 47fe3b51d11b43fbb68769d924f4ba26 to finish bootstrapping
I20240502 01:58:32.304988 28095 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 243a928cd9964e3bad0d41f297541754 to finish bootstrapping
I20240502 01:58:32.310983 28095 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7b98171c8198455aa773391897ea2b70 to finish bootstrapping
I20240502 01:58:32.316558 28095 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d206618059b843e8a77c4de818efdbc1 to finish bootstrapping
I20240502 01:58:32.321300 28095 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 432f171089fb4f67a46a5db2f9c2c610 to finish bootstrapping
I20240502 01:58:32.442994 348 heartbeater.cc:503] Master 127.27.111.254:35241 was elected leader, sending a full tablet report...
I20240502 01:58:32.517614 32750 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.518534 544 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.518596 413 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.519680 675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.520227 32613 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:32.613338 675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.613401 32613 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:32.613512 32750 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.614785 413 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.616463 544 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.620986 479 heartbeater.cc:503] Master 127.27.111.254:35241 was elected leader, sending a full tablet report...
I20240502 01:58:32.706660 765 consensus_queue.cc:1035] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d206618059b843e8a77c4de818efdbc1" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:32.707906 765 consensus_queue.cc:1035] T 5682d584bf8049ba95316340fada03bc P 47fe3b51d11b43fbb68769d924f4ba26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "432f171089fb4f67a46a5db2f9c2c610" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:32.844444 675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.844513 32750 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.844513 32613 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:32.845927 413 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.845943 544 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.937480 32750 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.937741 675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.940676 32613 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:32.943192 413 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.958367 544 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:33.051965 413 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:33.053012 675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:33.053579 32750 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:33.053958 544 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:33.063521 32613 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:33.093061 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 32557
I20240502 01:58:33.099434 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 32688
I20240502 01:58:33.105335 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 351
I20240502 01:58:33.111304 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 482
I20240502 01:58:33.117493 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 613
I20240502 01:58:33.123324 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 32466
2024-05-02T01:58:33Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (2312 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2312 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240502 01:58:33.146711 28095 test_util.cc:268] Using random seed: -255925934
I20240502 01:58:33.147523 28095 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:58:33.147567 28095 ts_itest-base.cc:113] --------------
I20240502 01:58:33.147578 28095 ts_itest-base.cc:114] 7 tablet servers
I20240502 01:58:33.147588 28095 ts_itest-base.cc:115] 2 replicas per TS
I20240502 01:58:33.147598 28095 ts_itest-base.cc:116] --------------
2024-05-02T01:58:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:58:33Z Disabled control of system clock
I20240502 01:58:33.155966 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.111.254:38251
--webserver_interface=127.27.111.254
--webserver_port=0
--builtin_ntp_servers=127.27.111.212:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.111.254:38251
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 01:58:33.266438 952 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 01:58:33.266719 952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:33.266747 952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:33.266791 952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:33.269842 952 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:33.269913 952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:58:33.269937 952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:33.269954 952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:58:33.269971 952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:58:33.273727 952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.111.254:38251
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.111.254:38251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.111.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:33.274699 952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:33.275858 952 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:33.284524 957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:33.284945 960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:33.285851 958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:33.286672 952 server_base.cc:1034] running on GCE node
I20240502 01:58:33.287273 952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:33.288297 952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:33.289453 952 hybrid_clock.cc:648] HybridClock initialized: now 1714615113289410 us; error 59 us; skew 500 ppm
I20240502 01:58:33.291558 952 webserver.cc:460] Webserver started at http://127.27.111.254:41741/ using document root <none> and password file <none>
I20240502 01:58:33.292137 952 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:33.292194 952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:33.292392 952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:33.294126 952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8cfd9cbfd05e40b09dbec2dce9761c79"
format_stamp: "Formatted at 2024-05-02 01:58:33 on dist-test-slave-0pw1"
I20240502 01:58:33.294782 952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8cfd9cbfd05e40b09dbec2dce9761c79"
format_stamp: "Formatted at 2024-05-02 01:58:33 on dist-test-slave-0pw1"
I20240502 01:58:33.298691 952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240502 01:58:33.300994 966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:33.302104 952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:33.302238 952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8cfd9cbfd05e40b09dbec2dce9761c79"
format_stamp: "Formatted at 2024-05-02 01:58:33 on dist-test-slave-0pw1"
I20240502 01:58:33.302335 952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:33.318727 952 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:33.319442 952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:33.319609 952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:33.326318 952 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.254:38251
I20240502 01:58:33.326328 1018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.254:38251 every 8 connection(s)
I20240502 01:58:33.327381 952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:58:33.330094 1019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:33.331599 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 952
I20240502 01:58:33.331717 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:58:33.336560 1019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79: Bootstrap starting.
I20240502 01:58:33.338924 1019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:33.339931 1019 log.cc:824] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:33.341822 1019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79: No bootstrap required, opened a new log
I20240502 01:58:33.344384 1019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd9cbfd05e40b09dbec2dce9761c79" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 38251 } }
I20240502 01:58:33.344585 1019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:33.344628 1019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cfd9cbfd05e40b09dbec2dce9761c79, State: Initialized, Role: FOLLOWER
I20240502 01:58:33.345038 1019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd9cbfd05e40b09dbec2dce9761c79" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 38251 } }
I20240502 01:58:33.345153 1019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:58:33.345196 1019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:58:33.345304 1019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:33.346100 1019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd9cbfd05e40b09dbec2dce9761c79" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 38251 } }
I20240502 01:58:33.346462 1019 leader_election.cc:302] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cfd9cbfd05e40b09dbec2dce9761c79; no voters:
I20240502 01:58:33.346787 1019 leader_election.cc:288] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:58:33.347256 1024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:33.347476 1024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [term 1 LEADER]: Becoming Leader. State: Replica: 8cfd9cbfd05e40b09dbec2dce9761c79, State: Running, Role: LEADER
I20240502 01:58:33.347812 1024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd9cbfd05e40b09dbec2dce9761c79" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 38251 } }
I20240502 01:58:33.348027 1019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:58:33.349349 1026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cfd9cbfd05e40b09dbec2dce9761c79. Latest consensus state: current_term: 1 leader_uuid: "8cfd9cbfd05e40b09dbec2dce9761c79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd9cbfd05e40b09dbec2dce9761c79" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 38251 } } }
I20240502 01:58:33.349475 1026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [sys.catalog]: This master's current role is: LEADER
I20240502 01:58:33.349779 1025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cfd9cbfd05e40b09dbec2dce9761c79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd9cbfd05e40b09dbec2dce9761c79" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 38251 } } }
I20240502 01:58:33.349870 1025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79 [sys.catalog]: This master's current role is: LEADER
W20240502 01:58:33.352679 1039 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 01:58:33.352758 1039 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 01:58:33.352897 1034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:58:33.353858 1034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:58:33.359289 1034 catalog_manager.cc:1347] Generated new cluster ID: f79d5107ce5d499a86346663dc303cb2
I20240502 01:58:33.359377 1034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:58:33.374008 1034 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:58:33.375396 1034 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:58:33.381479 1034 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8cfd9cbfd05e40b09dbec2dce9761c79: Generated new TSK 0
I20240502 01:58:33.382333 1034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:58:33.385592 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.193:0
--local_ip_for_outbound_sockets=127.27.111.193
--webserver_interface=127.27.111.193
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--builtin_ntp_servers=127.27.111.212:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:58:33.494706 1043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:33.494942 1043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:33.495016 1043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:33.498147 1043 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:33.498224 1043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:33.498317 1043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.193
I20240502 01:58:33.502003 1043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.111.193
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--never_fsync=true
--heap_profile_path=/tmp/kudu.1043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.193
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:33.502967 1043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:33.504108 1043 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:33.512885 1048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:33.513484 1051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:33.514053 1049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:33.514909 1043 server_base.cc:1034] running on GCE node
I20240502 01:58:33.515304 1043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:33.515864 1043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:33.517030 1043 hybrid_clock.cc:648] HybridClock initialized: now 1714615113517005 us; error 37 us; skew 500 ppm
I20240502 01:58:33.518888 1043 webserver.cc:460] Webserver started at http://127.27.111.193:38639/ using document root <none> and password file <none>
I20240502 01:58:33.519445 1043 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:33.519507 1043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:33.519685 1043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:33.521401 1043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a571e02bbe774f45b80bb4383642c683"
format_stamp: "Formatted at 2024-05-02 01:58:33 on dist-test-slave-0pw1"
I20240502 01:58:33.521865 1043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a571e02bbe774f45b80bb4383642c683"
format_stamp: "Formatted at 2024-05-02 01:58:33 on dist-test-slave-0pw1"
I20240502 01:58:33.525162 1043 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:58:33.527367 1057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:33.528355 1043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:58:33.528463 1043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a571e02bbe774f45b80bb4383642c683"
format_stamp: "Formatted at 2024-05-02 01:58:33 on dist-test-slave-0pw1"
I20240502 01:58:33.528554 1043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:33.557763 1043 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:33.558506 1043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:33.558669 1043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:33.559223 1043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:33.560149 1043 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:33.560201 1043 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:33.560262 1043 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:33.560293 1043 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:33.568562 1043 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.193:39473
I20240502 01:58:33.568588 1170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.193:39473 every 8 connection(s)
I20240502 01:58:33.569480 1043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:58:33.571749 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 1043
I20240502 01:58:33.571862 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:58:33.573945 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.194:0
--local_ip_for_outbound_sockets=127.27.111.194
--webserver_interface=127.27.111.194
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--builtin_ntp_servers=127.27.111.212:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:58:33.580408 1171 heartbeater.cc:347] Connected to a master server at 127.27.111.254:38251
I20240502 01:58:33.580659 1171 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:33.581115 1171 heartbeater.cc:511] Master 127.27.111.254:38251 requested a full tablet report, sending...
I20240502 01:58:33.582511 983 ts_manager.cc:194] Registered new tserver with Master: a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:33.583667 983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.193:56859
W20240502 01:58:33.691962 1174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:33.692205 1174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:33.692277 1174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:33.695379 1174 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:33.695466 1174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:33.695559 1174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.194
I20240502 01:58:33.699278 1174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.111.194
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--never_fsync=true
--heap_profile_path=/tmp/kudu.1174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.194
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:33.700199 1174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:33.701313 1174 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:33.709945 1179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:33.710433 1182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:33.711267 1180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:33.711946 1174 server_base.cc:1034] running on GCE node
I20240502 01:58:33.712318 1174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:33.712889 1174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:33.714040 1174 hybrid_clock.cc:648] HybridClock initialized: now 1714615113714027 us; error 26 us; skew 500 ppm
I20240502 01:58:33.715945 1174 webserver.cc:460] Webserver started at http://127.27.111.194:40757/ using document root <none> and password file <none>
I20240502 01:58:33.716507 1174 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:33.716571 1174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:33.716750 1174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:33.718494 1174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4d8c8df1c10b41fab826db1b2ef82e87"
format_stamp: "Formatted at 2024-05-02 01:58:33 on dist-test-slave-0pw1"
I20240502 01:58:33.718961 1174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4d8c8df1c10b41fab826db1b2ef82e87"
format_stamp: "Formatted at 2024-05-02 01:58:33 on dist-test-slave-0pw1"
I20240502 01:58:33.722293 1174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240502 01:58:33.724586 1188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:33.725615 1174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:33.725721 1174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4d8c8df1c10b41fab826db1b2ef82e87"
format_stamp: "Formatted at 2024-05-02 01:58:33 on dist-test-slave-0pw1"
I20240502 01:58:33.725814 1174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:33.745365 1174 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:33.746079 1174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:33.746255 1174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:33.746892 1174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:33.747850 1174 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:33.747910 1174 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:33.747974 1174 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:33.748013 1174 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:33.756820 1174 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.194:33267
I20240502 01:58:33.756841 1301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.194:33267 every 8 connection(s)
I20240502 01:58:33.757793 1174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:58:33.760020 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 1174
I20240502 01:58:33.760141 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:58:33.762584 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.195:0
--local_ip_for_outbound_sockets=127.27.111.195
--webserver_interface=127.27.111.195
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--builtin_ntp_servers=127.27.111.212:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:58:33.767552 1302 heartbeater.cc:347] Connected to a master server at 127.27.111.254:38251
I20240502 01:58:33.767781 1302 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:33.768232 1302 heartbeater.cc:511] Master 127.27.111.254:38251 requested a full tablet report, sending...
I20240502 01:58:33.769243 983 ts_manager.cc:194] Registered new tserver with Master: 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:33.770115 983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.194:32875
W20240502 01:58:33.876196 1305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:33.876451 1305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:33.876494 1305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:33.879530 1305 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:33.879593 1305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:33.879655 1305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.195
I20240502 01:58:33.883525 1305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.111.195
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--never_fsync=true
--heap_profile_path=/tmp/kudu.1305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.195
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:33.884443 1305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:33.885536 1305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:33.894196 1310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:33.894832 1311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:33.895558 1313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:33.896250 1305 server_base.cc:1034] running on GCE node
I20240502 01:58:33.896632 1305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:33.897207 1305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:33.898365 1305 hybrid_clock.cc:648] HybridClock initialized: now 1714615113898343 us; error 43 us; skew 500 ppm
I20240502 01:58:33.900211 1305 webserver.cc:460] Webserver started at http://127.27.111.195:35487/ using document root <none> and password file <none>
I20240502 01:58:33.900768 1305 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:33.900830 1305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:33.901044 1305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:33.902760 1305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3f073f261d9447b99220aa532c2b1072"
format_stamp: "Formatted at 2024-05-02 01:58:33 on dist-test-slave-0pw1"
I20240502 01:58:33.903241 1305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3f073f261d9447b99220aa532c2b1072"
format_stamp: "Formatted at 2024-05-02 01:58:33 on dist-test-slave-0pw1"
I20240502 01:58:33.906600 1305 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240502 01:58:33.908794 1319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:33.909804 1305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:33.909946 1305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3f073f261d9447b99220aa532c2b1072"
format_stamp: "Formatted at 2024-05-02 01:58:33 on dist-test-slave-0pw1"
I20240502 01:58:33.910037 1305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:33.932475 1305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:33.933140 1305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:33.933308 1305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:33.933858 1305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:33.934778 1305 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:33.934826 1305 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:33.934885 1305 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:33.934922 1305 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:33.943583 1305 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.195:45227
I20240502 01:58:33.943609 1432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.195:45227 every 8 connection(s)
I20240502 01:58:33.944535 1305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:58:33.949753 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 1305
I20240502 01:58:33.949856 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:58:33.951952 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.196:0
--local_ip_for_outbound_sockets=127.27.111.196
--webserver_interface=127.27.111.196
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--builtin_ntp_servers=127.27.111.212:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:58:33.954241 1433 heartbeater.cc:347] Connected to a master server at 127.27.111.254:38251
I20240502 01:58:33.954492 1433 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:33.954955 1433 heartbeater.cc:511] Master 127.27.111.254:38251 requested a full tablet report, sending...
I20240502 01:58:33.955964 983 ts_manager.cc:194] Registered new tserver with Master: 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:33.956815 983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.195:38537
W20240502 01:58:34.062373 1436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:34.062711 1436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:34.062788 1436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:34.065892 1436 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:34.065960 1436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:34.066056 1436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.196
I20240502 01:58:34.069665 1436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.111.196
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--never_fsync=true
--heap_profile_path=/tmp/kudu.1436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.196
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:34.070632 1436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:34.071749 1436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:34.080269 1441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:34.080997 1442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:34.081625 1444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:34.082355 1436 server_base.cc:1034] running on GCE node
I20240502 01:58:34.082830 1436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:34.083405 1436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:34.084566 1436 hybrid_clock.cc:648] HybridClock initialized: now 1714615114084555 us; error 31 us; skew 500 ppm
I20240502 01:58:34.086489 1436 webserver.cc:460] Webserver started at http://127.27.111.196:39957/ using document root <none> and password file <none>
I20240502 01:58:34.087085 1436 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:34.087147 1436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:34.087357 1436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:34.089040 1436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "09401a31cfd046a78a67a2d51e290224"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:34.089494 1436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "09401a31cfd046a78a67a2d51e290224"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:34.092819 1436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 01:58:34.095180 1450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:34.096338 1436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:58:34.096477 1436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "09401a31cfd046a78a67a2d51e290224"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:34.096576 1436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:34.129307 1436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:34.130096 1436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:34.130281 1436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:34.130937 1436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:34.131912 1436 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:34.131965 1436 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:34.132027 1436 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:34.132061 1436 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:34.140902 1436 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.196:43819
I20240502 01:58:34.140949 1563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.196:43819 every 8 connection(s)
I20240502 01:58:34.141911 1436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:58:34.148725 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 1436
I20240502 01:58:34.148844 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:58:34.150789 1564 heartbeater.cc:347] Connected to a master server at 127.27.111.254:38251
I20240502 01:58:34.151062 1564 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:34.151086 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.197:0
--local_ip_for_outbound_sockets=127.27.111.197
--webserver_interface=127.27.111.197
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--builtin_ntp_servers=127.27.111.212:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:58:34.151559 1564 heartbeater.cc:511] Master 127.27.111.254:38251 requested a full tablet report, sending...
I20240502 01:58:34.152585 983 ts_manager.cc:194] Registered new tserver with Master: 09401a31cfd046a78a67a2d51e290224 (127.27.111.196:43819)
I20240502 01:58:34.153131 983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.196:52521
W20240502 01:58:34.261958 1567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:34.262193 1567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:34.262265 1567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:34.265424 1567 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:34.265503 1567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:34.265606 1567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.197
I20240502 01:58:34.269232 1567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.111.197
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--never_fsync=true
--heap_profile_path=/tmp/kudu.1567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.197
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:34.270143 1567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:34.271535 1567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:34.280172 1572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:34.280680 1575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:34.281494 1573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:34.282276 1567 server_base.cc:1034] running on GCE node
I20240502 01:58:34.282732 1567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:34.283396 1567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:34.284629 1567 hybrid_clock.cc:648] HybridClock initialized: now 1714615114284563 us; error 44 us; skew 500 ppm
I20240502 01:58:34.286725 1567 webserver.cc:460] Webserver started at http://127.27.111.197:41999/ using document root <none> and password file <none>
I20240502 01:58:34.287277 1567 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:34.287334 1567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:34.287518 1567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:34.289181 1567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "07c427880bd54215847186eab221fce5"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:34.289642 1567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "07c427880bd54215847186eab221fce5"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:34.292887 1567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 01:58:34.295035 1581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:34.296028 1567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:34.296140 1567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "07c427880bd54215847186eab221fce5"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:34.296221 1567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:34.314740 1567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:34.315397 1567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:34.315547 1567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:34.316057 1567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:34.316931 1567 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:34.316980 1567 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:34.317044 1567 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:34.317071 1567 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:34.325254 1567 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.197:39309
I20240502 01:58:34.325305 1694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.197:39309 every 8 connection(s)
I20240502 01:58:34.326249 1567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:58:34.327157 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 1567
I20240502 01:58:34.327271 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:58:34.330260 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.198:0
--local_ip_for_outbound_sockets=127.27.111.198
--webserver_interface=127.27.111.198
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--builtin_ntp_servers=127.27.111.212:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:58:34.337018 1695 heartbeater.cc:347] Connected to a master server at 127.27.111.254:38251
I20240502 01:58:34.337296 1695 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:34.337936 1695 heartbeater.cc:511] Master 127.27.111.254:38251 requested a full tablet report, sending...
I20240502 01:58:34.339069 983 ts_manager.cc:194] Registered new tserver with Master: 07c427880bd54215847186eab221fce5 (127.27.111.197:39309)
I20240502 01:58:34.339604 983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.197:45575
W20240502 01:58:34.441259 1698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:34.441475 1698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:34.441550 1698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:34.444578 1698 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:34.444651 1698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:34.444748 1698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.198
I20240502 01:58:34.448525 1698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.111.198
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--never_fsync=true
--heap_profile_path=/tmp/kudu.1698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.198
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:34.449453 1698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:34.450623 1698 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:34.459156 1704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:34.459682 1706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:34.460534 1703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:34.461351 1698 server_base.cc:1034] running on GCE node
I20240502 01:58:34.461750 1698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:34.462349 1698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:34.463541 1698 hybrid_clock.cc:648] HybridClock initialized: now 1714615114463516 us; error 37 us; skew 500 ppm
I20240502 01:58:34.465471 1698 webserver.cc:460] Webserver started at http://127.27.111.198:39591/ using document root <none> and password file <none>
I20240502 01:58:34.466044 1698 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:34.466102 1698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:34.466288 1698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:34.468025 1698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "72de8d04b745434f923594c920aff90c"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:34.468479 1698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "72de8d04b745434f923594c920aff90c"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:34.471753 1698 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:58:34.473868 1712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:34.474884 1698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:34.474999 1698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "72de8d04b745434f923594c920aff90c"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:34.475091 1698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:34.509753 1698 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:34.510493 1698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:34.510659 1698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:34.511209 1698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:34.512094 1698 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:34.512141 1698 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:34.512185 1698 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:34.512216 1698 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:34.520620 1698 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.198:45287
I20240502 01:58:34.520643 1825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.198:45287 every 8 connection(s)
I20240502 01:58:34.521569 1698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:58:34.528915 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 1698
I20240502 01:58:34.529045 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:58:34.530045 1826 heartbeater.cc:347] Connected to a master server at 127.27.111.254:38251
I20240502 01:58:34.530248 1826 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:34.530786 1826 heartbeater.cc:511] Master 127.27.111.254:38251 requested a full tablet report, sending...
I20240502 01:58:34.531262 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.199:0
--local_ip_for_outbound_sockets=127.27.111.199
--webserver_interface=127.27.111.199
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--builtin_ntp_servers=127.27.111.212:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:58:34.531812 983 ts_manager.cc:194] Registered new tserver with Master: 72de8d04b745434f923594c920aff90c (127.27.111.198:45287)
I20240502 01:58:34.532330 983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.198:45713
I20240502 01:58:34.585759 1171 heartbeater.cc:503] Master 127.27.111.254:38251 was elected leader, sending a full tablet report...
W20240502 01:58:34.639674 1829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:34.639885 1829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:34.639930 1829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:34.642954 1829 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:34.643033 1829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:34.643110 1829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.199
I20240502 01:58:34.646710 1829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.111.199
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--never_fsync=true
--heap_profile_path=/tmp/kudu.1829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.199
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:34.647668 1829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:34.648779 1829 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:34.656966 1834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:34.657557 1837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:34.658476 1829 server_base.cc:1034] running on GCE node
W20240502 01:58:34.659045 1835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:34.659413 1829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:34.659976 1829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:34.661142 1829 hybrid_clock.cc:648] HybridClock initialized: now 1714615114661122 us; error 31 us; skew 500 ppm
I20240502 01:58:34.663127 1829 webserver.cc:460] Webserver started at http://127.27.111.199:43063/ using document root <none> and password file <none>
I20240502 01:58:34.663681 1829 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:34.663743 1829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:34.663960 1829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:58:34.665596 1829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "37279883fafe432f81fe7b8224dbe5fb"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:34.666103 1829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "37279883fafe432f81fe7b8224dbe5fb"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:34.669456 1829 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 01:58:34.671563 1843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:34.672643 1829 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:34.672739 1829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "37279883fafe432f81fe7b8224dbe5fb"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:34.672814 1829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:34.682529 1829 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:34.683105 1829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:34.683225 1829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:34.683676 1829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:34.684520 1829 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:34.684573 1829 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:34.684617 1829 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:34.684639 1829 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:34.692628 1829 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.199:33361
I20240502 01:58:34.692653 1956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.199:33361 every 8 connection(s)
I20240502 01:58:34.693480 1829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:58:34.697975 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 1829
I20240502 01:58:34.698073 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:58:34.702487 1957 heartbeater.cc:347] Connected to a master server at 127.27.111.254:38251
I20240502 01:58:34.702708 1957 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:34.703177 1957 heartbeater.cc:511] Master 127.27.111.254:38251 requested a full tablet report, sending...
I20240502 01:58:34.704116 983 ts_manager.cc:194] Registered new tserver with Master: 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199:33361)
I20240502 01:58:34.704624 983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.199:34961
I20240502 01:58:34.712218 28095 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 01:58:34.720834 983 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34484:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 01:58:34.736709 1629 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa2250275ba4eb391f88f9ee0fd52e9 (DEFAULT_TABLE table=TestTable [id=0228923519bb4765b2400b84da3b512d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.736713 1105 tablet_service.cc:1467] Processing CreateTablet for tablet 0fa2250275ba4eb391f88f9ee0fd52e9 (DEFAULT_TABLE table=TestTable [id=0228923519bb4765b2400b84da3b512d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:34.737540 1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa2250275ba4eb391f88f9ee0fd52e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.737542 1105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fa2250275ba4eb391f88f9ee0fd52e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:34.742208 1973 tablet_bootstrap.cc:492] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683: Bootstrap starting.
I20240502 01:58:34.742507 1974 tablet_bootstrap.cc:492] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5: Bootstrap starting.
I20240502 01:58:34.744203 1974 tablet_bootstrap.cc:654] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.744204 1973 tablet_bootstrap.cc:654] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:34.744839 1974 log.cc:824] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:34.744839 1973 log.cc:824] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:34.746347 1974 tablet_bootstrap.cc:492] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5: No bootstrap required, opened a new log
I20240502 01:58:34.746383 1973 tablet_bootstrap.cc:492] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683: No bootstrap required, opened a new log
I20240502 01:58:34.746548 1974 ts_tablet_manager.cc:1367] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240502 01:58:34.746558 1973 ts_tablet_manager.cc:1367] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:58:34.749467 1974 raft_consensus.cc:348] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:34.749637 1974 raft_consensus.cc:374] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.749526 1973 raft_consensus.cc:348] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:34.749677 1974 raft_consensus.cc:729] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c427880bd54215847186eab221fce5, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.749686 1973 raft_consensus.cc:374] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:34.749744 1973 raft_consensus.cc:729] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Initialized, Role: FOLLOWER
I20240502 01:58:34.750089 1974 consensus_queue.cc:260] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:34.750133 1973 consensus_queue.cc:260] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:34.750963 1973 ts_tablet_manager.cc:1398] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:58:34.750982 1974 ts_tablet_manager.cc:1398] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20240502 01:58:34.751315 1695 heartbeater.cc:503] Master 127.27.111.254:38251 was elected leader, sending a full tablet report...
I20240502 01:58:34.772087 1302 heartbeater.cc:503] Master 127.27.111.254:38251 was elected leader, sending a full tablet report...
W20240502 01:58:34.821161 1172 tablet.cc:2369] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:58:34.827323 1696 tablet.cc:2369] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:34.946938 1977 raft_consensus.cc:482] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:34.947124 1977 raft_consensus.cc:504] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:34.948141 1977 leader_election.cc:288] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:34.951954 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa2250275ba4eb391f88f9ee0fd52e9" candidate_uuid: "07c427880bd54215847186eab221fce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a571e02bbe774f45b80bb4383642c683" is_pre_election: true
I20240502 01:58:34.952250 1125 raft_consensus.cc:2454] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c427880bd54215847186eab221fce5 in term 0.
I20240502 01:58:34.952719 1583 leader_election.cc:302] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 07c427880bd54215847186eab221fce5, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:34.953074 1977 raft_consensus.cc:2781] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:34.953166 1977 raft_consensus.cc:482] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:34.953219 1977 raft_consensus.cc:3037] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.954140 1977 raft_consensus.cc:504] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:34.954444 1977 leader_election.cc:288] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [CANDIDATE]: Term 1 election: Requested vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:34.954828 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa2250275ba4eb391f88f9ee0fd52e9" candidate_uuid: "07c427880bd54215847186eab221fce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a571e02bbe774f45b80bb4383642c683"
I20240502 01:58:34.954954 1125 raft_consensus.cc:3037] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:34.955839 1125 raft_consensus.cc:2454] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c427880bd54215847186eab221fce5 in term 1.
I20240502 01:58:34.956161 1583 leader_election.cc:302] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 07c427880bd54215847186eab221fce5, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:34.956315 1977 raft_consensus.cc:2781] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:34.956509 1977 raft_consensus.cc:686] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 1 LEADER]: Becoming Leader. State: Replica: 07c427880bd54215847186eab221fce5, State: Running, Role: LEADER
I20240502 01:58:34.956833 1977 consensus_queue.cc:237] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:34.958842 1433 heartbeater.cc:503] Master 127.27.111.254:38251 was elected leader, sending a full tablet report...
I20240502 01:58:34.959723 983 catalog_manager.cc:5484] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07c427880bd54215847186eab221fce5 (127.27.111.197). New cstate: current_term: 1 leader_uuid: "07c427880bd54215847186eab221fce5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:35.022136 28095 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 01:58:35.023612 28095 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a571e02bbe774f45b80bb4383642c683 to finish bootstrapping
I20240502 01:58:35.028982 28095 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4d8c8df1c10b41fab826db1b2ef82e87 to finish bootstrapping
I20240502 01:58:35.035094 28095 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3f073f261d9447b99220aa532c2b1072 to finish bootstrapping
I20240502 01:58:35.040601 28095 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 09401a31cfd046a78a67a2d51e290224 to finish bootstrapping
I20240502 01:58:35.046490 28095 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 07c427880bd54215847186eab221fce5 to finish bootstrapping
I20240502 01:58:35.051415 28095 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 72de8d04b745434f923594c920aff90c to finish bootstrapping
I20240502 01:58:35.056952 28095 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 37279883fafe432f81fe7b8224dbe5fb to finish bootstrapping
I20240502 01:58:35.062776 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 1436
I20240502 01:58:35.068727 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 1567
I20240502 01:58:35.076263 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 1698
I20240502 01:58:35.081856 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 1829
I20240502 01:58:36.835177 1988 raft_consensus.cc:482] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:36.835340 1988 raft_consensus.cc:504] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:36.836375 1988 leader_election.cc:288] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07c427880bd54215847186eab221fce5 (127.27.111.197:39309)
W20240502 01:58:36.837090 1061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.197:39309: connect: Connection refused (error 111)
W20240502 01:58:36.837980 1061 leader_election.cc:334] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 07c427880bd54215847186eab221fce5 (127.27.111.197:39309): Network error: Client connection negotiation failed: client connection to 127.27.111.197:39309: connect: Connection refused (error 111)
I20240502 01:58:36.838069 1061 leader_election.cc:302] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a571e02bbe774f45b80bb4383642c683; no voters: 07c427880bd54215847186eab221fce5
I20240502 01:58:36.838279 1988 raft_consensus.cc:2726] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:58:38.356751 1991 raft_consensus.cc:482] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:38.356910 1991 raft_consensus.cc:504] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:38.357263 1991 leader_election.cc:288] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07c427880bd54215847186eab221fce5 (127.27.111.197:39309)
W20240502 01:58:38.358642 1061 leader_election.cc:334] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 07c427880bd54215847186eab221fce5 (127.27.111.197:39309): Network error: Client connection negotiation failed: client connection to 127.27.111.197:39309: connect: Connection refused (error 111)
I20240502 01:58:38.358721 1061 leader_election.cc:302] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a571e02bbe774f45b80bb4383642c683; no voters: 07c427880bd54215847186eab221fce5
I20240502 01:58:38.358875 1991 raft_consensus.cc:2726] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:58:38.838806 983 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34484:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:38.846566 1105 tablet_service.cc:1467] Processing CreateTablet for tablet 609e71d9a50b4497a0410eb09734c169 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a79d488e199a471f817d760819b107cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.846915 1105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609e71d9a50b4497a0410eb09734c169. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.849390 2002 tablet_bootstrap.cc:492] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683: Bootstrap starting.
I20240502 01:58:38.850517 2002 tablet_bootstrap.cc:654] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.852115 1236 tablet_service.cc:1467] Processing CreateTablet for tablet 844edd610a734b0994f59a4b0e29af0b (DEFAULT_TABLE table=rebalance_test_table_0 [id=a79d488e199a471f817d760819b107cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.852108 1367 tablet_service.cc:1467] Processing CreateTablet for tablet 844edd610a734b0994f59a4b0e29af0b (DEFAULT_TABLE table=rebalance_test_table_0 [id=a79d488e199a471f817d760819b107cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.853097 1236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844edd610a734b0994f59a4b0e29af0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.853097 1367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844edd610a734b0994f59a4b0e29af0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.854594 1366 tablet_service.cc:1467] Processing CreateTablet for tablet 82337220b48d45fba654828a31eabbf4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a79d488e199a471f817d760819b107cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.854888 1366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82337220b48d45fba654828a31eabbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.855831 2002 tablet_bootstrap.cc:492] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683: No bootstrap required, opened a new log
I20240502 01:58:38.855840 1235 tablet_service.cc:1467] Processing CreateTablet for tablet 82337220b48d45fba654828a31eabbf4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a79d488e199a471f817d760819b107cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.855931 2002 ts_tablet_manager.cc:1367] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 01:58:38.856107 1235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82337220b48d45fba654828a31eabbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.856540 2002 raft_consensus.cc:348] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:38.856652 2002 raft_consensus.cc:374] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.856681 2002 raft_consensus.cc:729] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.856812 2002 consensus_queue.cc:260] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:38.857070 2002 ts_tablet_manager.cc:1398] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:38.858000 1234 tablet_service.cc:1467] Processing CreateTablet for tablet 609e71d9a50b4497a0410eb09734c169 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a79d488e199a471f817d760819b107cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:38.858235 1234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609e71d9a50b4497a0410eb09734c169. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:38.859648 2005 tablet_bootstrap.cc:492] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072: Bootstrap starting.
I20240502 01:58:38.860443 2004 tablet_bootstrap.cc:492] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87: Bootstrap starting.
I20240502 01:58:38.862301 2004 tablet_bootstrap.cc:654] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.862862 2005 tablet_bootstrap.cc:654] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.863186 2004 log.cc:824] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:38.863451 2005 log.cc:824] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:38.864775 2004 tablet_bootstrap.cc:492] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87: No bootstrap required, opened a new log
I20240502 01:58:38.864775 2005 tablet_bootstrap.cc:492] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072: No bootstrap required, opened a new log
I20240502 01:58:38.864948 2005 ts_tablet_manager.cc:1367] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:58:38.864948 2004 ts_tablet_manager.cc:1367] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:58:38.868330 2004 raft_consensus.cc:348] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:38.868541 2004 raft_consensus.cc:374] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.868599 2004 raft_consensus.cc:729] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.868597 2005 raft_consensus.cc:348] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:38.868760 2005 raft_consensus.cc:374] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.868818 2005 raft_consensus.cc:729] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f073f261d9447b99220aa532c2b1072, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.869061 2004 consensus_queue.cc:260] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:38.869141 2005 consensus_queue.cc:260] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:38.871227 2004 ts_tablet_manager.cc:1398] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 01:58:38.871441 2004 tablet_bootstrap.cc:492] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87: Bootstrap starting.
I20240502 01:58:38.872099 2005 ts_tablet_manager.cc:1398] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:58:38.872325 2005 tablet_bootstrap.cc:492] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072: Bootstrap starting.
I20240502 01:58:38.872361 2004 tablet_bootstrap.cc:654] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.872678 2009 raft_consensus.cc:482] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:38.872884 2009 raft_consensus.cc:504] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:38.873558 2005 tablet_bootstrap.cc:654] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.873962 2009 leader_election.cc:288] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:38.874531 2004 tablet_bootstrap.cc:492] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87: No bootstrap required, opened a new log
I20240502 01:58:38.874612 2004 ts_tablet_manager.cc:1367] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:38.874914 2005 tablet_bootstrap.cc:492] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072: No bootstrap required, opened a new log
I20240502 01:58:38.874980 2005 ts_tablet_manager.cc:1367] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:38.875151 2004 raft_consensus.cc:348] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:38.875254 2004 raft_consensus.cc:374] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.875295 2004 raft_consensus.cc:729] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.875439 2004 consensus_queue.cc:260] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:38.875455 2005 raft_consensus.cc:348] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:38.875552 2005 raft_consensus.cc:374] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.875586 2005 raft_consensus.cc:729] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f073f261d9447b99220aa532c2b1072, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.875705 2005 consensus_queue.cc:260] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:38.876073 2004 ts_tablet_manager.cc:1398] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:38.876220 2004 tablet_bootstrap.cc:492] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87: Bootstrap starting.
I20240502 01:58:38.876211 2005 ts_tablet_manager.cc:1398] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:38.877110 2004 tablet_bootstrap.cc:654] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:38.878755 2004 tablet_bootstrap.cc:492] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87: No bootstrap required, opened a new log
I20240502 01:58:38.878844 2004 ts_tablet_manager.cc:1367] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:58:38.879192 2004 raft_consensus.cc:348] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:38.879292 2004 raft_consensus.cc:374] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:38.879320 2004 raft_consensus.cc:729] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Initialized, Role: FOLLOWER
I20240502 01:58:38.879354 1256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82337220b48d45fba654828a31eabbf4" candidate_uuid: "3f073f261d9447b99220aa532c2b1072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" is_pre_election: true
I20240502 01:58:38.879448 2004 consensus_queue.cc:260] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:38.879603 1256 raft_consensus.cc:2454] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f073f261d9447b99220aa532c2b1072 in term 0.
I20240502 01:58:38.879844 2004 ts_tablet_manager.cc:1398] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:38.880059 1320 leader_election.cc:302] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, 4d8c8df1c10b41fab826db1b2ef82e87; no voters:
I20240502 01:58:38.880275 2009 raft_consensus.cc:2781] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:38.880364 2009 raft_consensus.cc:482] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:38.880421 2009 raft_consensus.cc:3037] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:38.881459 2009 raft_consensus.cc:504] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:38.881736 2009 leader_election.cc:288] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [CANDIDATE]: Term 1 election: Requested vote from peers 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:38.882081 1256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82337220b48d45fba654828a31eabbf4" candidate_uuid: "3f073f261d9447b99220aa532c2b1072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d8c8df1c10b41fab826db1b2ef82e87"
I20240502 01:58:38.882202 1256 raft_consensus.cc:3037] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:38.883070 1256 raft_consensus.cc:2454] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f073f261d9447b99220aa532c2b1072 in term 1.
I20240502 01:58:38.883347 1320 leader_election.cc:302] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, 4d8c8df1c10b41fab826db1b2ef82e87; no voters:
I20240502 01:58:38.883507 2009 raft_consensus.cc:2781] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:38.883698 2009 raft_consensus.cc:686] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [term 1 LEADER]: Becoming Leader. State: Replica: 3f073f261d9447b99220aa532c2b1072, State: Running, Role: LEADER
I20240502 01:58:38.883970 2009 consensus_queue.cc:237] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:38.886437 983 catalog_manager.cc:5484] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f073f261d9447b99220aa532c2b1072 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "3f073f261d9447b99220aa532c2b1072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } }
W20240502 01:58:38.949468 1434 tablet.cc:2369] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:58:39.012483 1303 tablet.cc:2369] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:58:39.060329 2008 raft_consensus.cc:482] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:39.060484 2008 raft_consensus.cc:504] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:39.061446 2008 leader_election.cc:288] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:39.064993 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609e71d9a50b4497a0410eb09734c169" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a571e02bbe774f45b80bb4383642c683" is_pre_election: true
I20240502 01:58:39.065135 1125 raft_consensus.cc:2454] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 0.
I20240502 01:58:39.065485 1190 leader_election.cc:302] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d8c8df1c10b41fab826db1b2ef82e87, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:39.065726 2008 raft_consensus.cc:2781] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:39.065795 2008 raft_consensus.cc:482] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:39.065876 2008 raft_consensus.cc:3037] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:39.066746 2008 raft_consensus.cc:504] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:39.067065 2008 leader_election.cc:288] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Requested vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:39.067445 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609e71d9a50b4497a0410eb09734c169" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a571e02bbe774f45b80bb4383642c683"
I20240502 01:58:39.067548 1125 raft_consensus.cc:3037] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:39.068310 1125 raft_consensus.cc:2454] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 1.
I20240502 01:58:39.068606 1190 leader_election.cc:302] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d8c8df1c10b41fab826db1b2ef82e87, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:39.068759 2008 raft_consensus.cc:2781] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:39.068982 2008 raft_consensus.cc:686] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Becoming Leader. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Running, Role: LEADER
I20240502 01:58:39.069278 2008 consensus_queue.cc:237] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:39.072083 983 catalog_manager.cc:5484] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } }
I20240502 01:58:39.088094 2008 raft_consensus.cc:482] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:39.088240 2008 raft_consensus.cc:504] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.088666 2008 leader_election.cc:288] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:39.091848 1387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844edd610a734b0994f59a4b0e29af0b" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f073f261d9447b99220aa532c2b1072" is_pre_election: true
I20240502 01:58:39.092053 1387 raft_consensus.cc:2454] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 0.
I20240502 01:58:39.092377 1190 leader_election.cc:302] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, 4d8c8df1c10b41fab826db1b2ef82e87; no voters:
I20240502 01:58:39.092559 2008 raft_consensus.cc:2781] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:39.092623 2008 raft_consensus.cc:482] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:39.092677 2008 raft_consensus.cc:3037] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:39.093475 2008 raft_consensus.cc:504] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.093760 2008 leader_election.cc:288] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Requested vote from peers 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:39.094184 1387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844edd610a734b0994f59a4b0e29af0b" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f073f261d9447b99220aa532c2b1072"
I20240502 01:58:39.094296 1387 raft_consensus.cc:3037] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:39.095185 1387 raft_consensus.cc:2454] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 1.
I20240502 01:58:39.095484 1190 leader_election.cc:302] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, 4d8c8df1c10b41fab826db1b2ef82e87; no voters:
I20240502 01:58:39.095645 2008 raft_consensus.cc:2781] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:39.095743 2008 raft_consensus.cc:686] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Becoming Leader. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Running, Role: LEADER
I20240502 01:58:39.095880 2008 consensus_queue.cc:237] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.097321 983 catalog_manager.cc:5484] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } }
W20240502 01:58:39.249006 2021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:39.249300 2021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:39.252449 2021 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 01:58:39.333917 2009 consensus_queue.cc:1035] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:39.356070 1125 raft_consensus.cc:1261] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:39.356756 2008 consensus_queue.cc:1035] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:39.361399 1387 raft_consensus.cc:1261] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:39.361877 2008 consensus_queue.cc:1035] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:39.366514 2054 mvcc.cc:205] Tried to move back new op lower bound from 7023063528876163072 to 7023063527817621504. Current Snapshot: MvccSnapshot[applied={T|T < 7023063528876163072}]
I20240502 01:58:39.367475 2047 mvcc.cc:205] Tried to move back new op lower bound from 7023063528876163072 to 7023063527817621504. Current Snapshot: MvccSnapshot[applied={T|T < 7023063528876163072}]
I20240502 01:58:39.374541 2058 mvcc.cc:205] Tried to move back new op lower bound from 7023063528876199936 to 7023063527709704192. Current Snapshot: MvccSnapshot[applied={T|T < 7023063528876199936}]
INSERT report
rows total: 2000
time total: 102.342 ms
time per row: 0.051171 ms
I20240502 01:58:39.403620 977 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34484:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:39.408912 1105 tablet_service.cc:1467] Processing CreateTablet for tablet 5bba25f4abe945429b0cc8ac93e12c8a (DEFAULT_TABLE table=rebalance_test_table_1 [id=e6f3958b51cd48bfb0764c9b91209835]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:39.408954 1367 tablet_service.cc:1467] Processing CreateTablet for tablet 5bba25f4abe945429b0cc8ac93e12c8a (DEFAULT_TABLE table=rebalance_test_table_1 [id=e6f3958b51cd48bfb0764c9b91209835]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:39.409247 1105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bba25f4abe945429b0cc8ac93e12c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.409256 1367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bba25f4abe945429b0cc8ac93e12c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.409319 1366 tablet_service.cc:1467] Processing CreateTablet for tablet d0cde29604ac4292be1ead0a69aa6908 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e6f3958b51cd48bfb0764c9b91209835]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:39.409516 1366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cde29604ac4292be1ead0a69aa6908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.410013 1104 tablet_service.cc:1467] Processing CreateTablet for tablet d0cde29604ac4292be1ead0a69aa6908 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e6f3958b51cd48bfb0764c9b91209835]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:39.410251 1104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cde29604ac4292be1ead0a69aa6908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.412195 2069 tablet_bootstrap.cc:492] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683: Bootstrap starting.
I20240502 01:58:39.412274 1236 tablet_service.cc:1467] Processing CreateTablet for tablet c0e16afa25404fa78e03547f7b140283 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e6f3958b51cd48bfb0764c9b91209835]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:39.412515 1236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e16afa25404fa78e03547f7b140283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.412508 1103 tablet_service.cc:1467] Processing CreateTablet for tablet c0e16afa25404fa78e03547f7b140283 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e6f3958b51cd48bfb0764c9b91209835]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:39.412724 1103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e16afa25404fa78e03547f7b140283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:39.413230 2069 tablet_bootstrap.cc:654] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:39.414294 2070 tablet_bootstrap.cc:492] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072: Bootstrap starting.
I20240502 01:58:39.415210 2069 tablet_bootstrap.cc:492] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683: No bootstrap required, opened a new log
I20240502 01:58:39.415210 2072 tablet_bootstrap.cc:492] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87: Bootstrap starting.
I20240502 01:58:39.415304 2069 ts_tablet_manager.cc:1367] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:39.415458 2070 tablet_bootstrap.cc:654] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:39.415757 2069 raft_consensus.cc:348] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.415884 2069 raft_consensus.cc:374] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:39.415920 2069 raft_consensus.cc:729] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Initialized, Role: FOLLOWER
I20240502 01:58:39.416055 2072 tablet_bootstrap.cc:654] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:39.416078 2069 consensus_queue.cc:260] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.416479 2070 tablet_bootstrap.cc:492] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072: No bootstrap required, opened a new log
I20240502 01:58:39.416497 2069 ts_tablet_manager.cc:1398] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:39.416558 2070 ts_tablet_manager.cc:1367] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:39.416661 2069 tablet_bootstrap.cc:492] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683: Bootstrap starting.
I20240502 01:58:39.417028 2070 raft_consensus.cc:348] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.417120 2070 raft_consensus.cc:374] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:39.417150 2070 raft_consensus.cc:729] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f073f261d9447b99220aa532c2b1072, State: Initialized, Role: FOLLOWER
I20240502 01:58:39.417263 2070 consensus_queue.cc:260] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.417526 2070 ts_tablet_manager.cc:1398] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:39.417690 2070 tablet_bootstrap.cc:492] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072: Bootstrap starting.
I20240502 01:58:39.417838 2069 tablet_bootstrap.cc:654] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:39.418586 2070 tablet_bootstrap.cc:654] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:39.419175 2069 tablet_bootstrap.cc:492] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683: No bootstrap required, opened a new log
I20240502 01:58:39.419265 2069 ts_tablet_manager.cc:1367] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:39.419463 2072 tablet_bootstrap.cc:492] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87: No bootstrap required, opened a new log
I20240502 01:58:39.419535 2072 ts_tablet_manager.cc:1367] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:39.419687 2069 raft_consensus.cc:348] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.419775 2069 raft_consensus.cc:374] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:39.419809 2069 raft_consensus.cc:729] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Initialized, Role: FOLLOWER
I20240502 01:58:39.419930 2069 consensus_queue.cc:260] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.419965 2072 raft_consensus.cc:348] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:39.420045 2072 raft_consensus.cc:374] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:39.420074 2072 raft_consensus.cc:729] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Initialized, Role: FOLLOWER
I20240502 01:58:39.420192 2069 ts_tablet_manager.cc:1398] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:39.420181 2072 consensus_queue.cc:260] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:39.420332 2069 tablet_bootstrap.cc:492] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683: Bootstrap starting.
I20240502 01:58:39.420433 2072 ts_tablet_manager.cc:1398] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:39.420536 2070 tablet_bootstrap.cc:492] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072: No bootstrap required, opened a new log
I20240502 01:58:39.420609 2070 ts_tablet_manager.cc:1367] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:39.421001 2070 raft_consensus.cc:348] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.421103 2070 raft_consensus.cc:374] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:39.421139 2070 raft_consensus.cc:729] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f073f261d9447b99220aa532c2b1072, State: Initialized, Role: FOLLOWER
I20240502 01:58:39.421180 2069 tablet_bootstrap.cc:654] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:39.421247 2070 consensus_queue.cc:260] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.421515 2070 ts_tablet_manager.cc:1398] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:39.422775 2069 tablet_bootstrap.cc:492] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683: No bootstrap required, opened a new log
I20240502 01:58:39.422886 2069 ts_tablet_manager.cc:1367] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:39.423267 2069 raft_consensus.cc:348] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:39.423363 2069 raft_consensus.cc:374] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:39.423395 2069 raft_consensus.cc:729] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Initialized, Role: FOLLOWER
I20240502 01:58:39.423532 2069 consensus_queue.cc:260] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:39.423792 2069 ts_tablet_manager.cc:1398] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:39.663766 2056 raft_consensus.cc:482] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:39.663913 2056 raft_consensus.cc:504] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.664348 2056 leader_election.cc:288] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:39.667914 1387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cde29604ac4292be1ead0a69aa6908" candidate_uuid: "a571e02bbe774f45b80bb4383642c683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f073f261d9447b99220aa532c2b1072" is_pre_election: true
I20240502 01:58:39.668063 1387 raft_consensus.cc:2454] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a571e02bbe774f45b80bb4383642c683 in term 0.
I20240502 01:58:39.668406 1059 leader_election.cc:302] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:39.668618 2056 raft_consensus.cc:2781] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:39.668677 2056 raft_consensus.cc:482] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:39.668706 2056 raft_consensus.cc:3037] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:39.669584 2056 raft_consensus.cc:504] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.669857 2056 leader_election.cc:288] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 election: Requested vote from peers 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:39.670190 1387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cde29604ac4292be1ead0a69aa6908" candidate_uuid: "a571e02bbe774f45b80bb4383642c683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f073f261d9447b99220aa532c2b1072"
I20240502 01:58:39.670289 1387 raft_consensus.cc:3037] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:39.671124 1387 raft_consensus.cc:2454] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a571e02bbe774f45b80bb4383642c683 in term 1.
I20240502 01:58:39.671407 1059 leader_election.cc:302] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:39.671581 2056 raft_consensus.cc:2781] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:39.671768 2056 raft_consensus.cc:686] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Becoming Leader. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Running, Role: LEADER
I20240502 01:58:39.672077 2056 consensus_queue.cc:237] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.674068 977 catalog_manager.cc:5484] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 reported cstate change: term changed from 0 to 1, leader changed from <none> to a571e02bbe774f45b80bb4383642c683 (127.27.111.193). New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } }
I20240502 01:58:39.711918 2065 raft_consensus.cc:482] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:39.712054 2065 raft_consensus.cc:504] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:39.712399 2065 leader_election.cc:288] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:39.712841 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e16afa25404fa78e03547f7b140283" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a571e02bbe774f45b80bb4383642c683" is_pre_election: true
I20240502 01:58:39.712978 1125 raft_consensus.cc:2454] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 0.
I20240502 01:58:39.713240 1190 leader_election.cc:302] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d8c8df1c10b41fab826db1b2ef82e87, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:39.713419 2065 raft_consensus.cc:2781] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:39.713486 2065 raft_consensus.cc:482] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:39.713531 2065 raft_consensus.cc:3037] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:39.714444 2065 raft_consensus.cc:504] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:39.714692 2065 leader_election.cc:288] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Requested vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:39.715047 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e16afa25404fa78e03547f7b140283" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a571e02bbe774f45b80bb4383642c683"
I20240502 01:58:39.715142 1125 raft_consensus.cc:3037] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:39.715870 1125 raft_consensus.cc:2454] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 1.
I20240502 01:58:39.716147 1190 leader_election.cc:302] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d8c8df1c10b41fab826db1b2ef82e87, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:39.716297 2065 raft_consensus.cc:2781] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:39.716372 2065 raft_consensus.cc:686] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Becoming Leader. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Running, Role: LEADER
I20240502 01:58:39.716498 2065 consensus_queue.cc:237] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:39.717975 977 catalog_manager.cc:5484] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } }
I20240502 01:58:39.771685 2056 raft_consensus.cc:482] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:39.771822 2056 raft_consensus.cc:504] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.772188 2056 leader_election.cc:288] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:39.772667 1387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bba25f4abe945429b0cc8ac93e12c8a" candidate_uuid: "a571e02bbe774f45b80bb4383642c683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f073f261d9447b99220aa532c2b1072" is_pre_election: true
I20240502 01:58:39.772801 1387 raft_consensus.cc:2454] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a571e02bbe774f45b80bb4383642c683 in term 0.
I20240502 01:58:39.773072 1059 leader_election.cc:302] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:39.773267 2056 raft_consensus.cc:2781] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:39.773334 2056 raft_consensus.cc:482] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:39.773392 2056 raft_consensus.cc:3037] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:39.774245 2056 raft_consensus.cc:504] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.774542 2056 leader_election.cc:288] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 election: Requested vote from peers 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:39.774875 1387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bba25f4abe945429b0cc8ac93e12c8a" candidate_uuid: "a571e02bbe774f45b80bb4383642c683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f073f261d9447b99220aa532c2b1072"
I20240502 01:58:39.774992 1387 raft_consensus.cc:3037] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:39.775893 1387 raft_consensus.cc:2454] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a571e02bbe774f45b80bb4383642c683 in term 1.
I20240502 01:58:39.776183 1059 leader_election.cc:302] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:39.776366 2056 raft_consensus.cc:2781] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:39.776446 2056 raft_consensus.cc:686] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Becoming Leader. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Running, Role: LEADER
I20240502 01:58:39.776574 2056 consensus_queue.cc:237] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:39.778151 977 catalog_manager.cc:5484] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 reported cstate change: term changed from 0 to 1, leader changed from <none> to a571e02bbe774f45b80bb4383642c683 (127.27.111.193). New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } }
W20240502 01:58:39.972865 2083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:39.973098 2083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:39.976053 2083 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 01:58:40.060253 1387 raft_consensus.cc:1261] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:40.060721 1386 raft_consensus.cc:1261] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:40.061110 2056 consensus_queue.cc:1035] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:40.062899 2056 consensus_queue.cc:1035] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:40.068408 1125 raft_consensus.cc:1261] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:40.069062 2065 consensus_queue.cc:1035] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 82.7642 ms
time per row: 0.0413821 ms
I20240502 01:58:40.103097 974 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34484:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:40.108645 1236 tablet_service.cc:1467] Processing CreateTablet for tablet 299019dc3d0848ad9324a84e319b9222 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2fba41bfdcb948309c7c7ecff39e9c06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.108788 1234 tablet_service.cc:1467] Processing CreateTablet for tablet c8a946d0b927414f88c615d54ce5fc7f (DEFAULT_TABLE table=rebalance_test_table_2 [id=2fba41bfdcb948309c7c7ecff39e9c06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.108987 1236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 299019dc3d0848ad9324a84e319b9222. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.108983 1366 tablet_service.cc:1467] Processing CreateTablet for tablet 299019dc3d0848ad9324a84e319b9222 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2fba41bfdcb948309c7c7ecff39e9c06]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.109025 1367 tablet_service.cc:1467] Processing CreateTablet for tablet c8a946d0b927414f88c615d54ce5fc7f (DEFAULT_TABLE table=rebalance_test_table_2 [id=2fba41bfdcb948309c7c7ecff39e9c06]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.109228 1234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a946d0b927414f88c615d54ce5fc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.109246 1366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 299019dc3d0848ad9324a84e319b9222. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.109409 1103 tablet_service.cc:1467] Processing CreateTablet for tablet fa93c869b01b4af69678e378df7db684 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2fba41bfdcb948309c7c7ecff39e9c06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.109679 1103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa93c869b01b4af69678e378df7db684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.111464 2130 tablet_bootstrap.cc:492] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072: Bootstrap starting.
I20240502 01:58:40.111809 2131 tablet_bootstrap.cc:492] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87: Bootstrap starting.
I20240502 01:58:40.112234 1367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a946d0b927414f88c615d54ce5fc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.112752 2131 tablet_bootstrap.cc:654] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.112795 1234 tablet_service.cc:1467] Processing CreateTablet for tablet fa93c869b01b4af69678e378df7db684 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2fba41bfdcb948309c7c7ecff39e9c06]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.113046 1234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa93c869b01b4af69678e378df7db684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.113831 2131 tablet_bootstrap.cc:492] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87: No bootstrap required, opened a new log
I20240502 01:58:40.113924 2131 ts_tablet_manager.cc:1367] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:40.112124 2132 tablet_bootstrap.cc:492] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683: Bootstrap starting.
I20240502 01:58:40.114445 2131 raft_consensus.cc:348] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.114569 2131 raft_consensus.cc:374] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.114604 2131 raft_consensus.cc:729] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.114758 2131 consensus_queue.cc:260] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.115284 2132 tablet_bootstrap.cc:654] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.115512 2131 ts_tablet_manager.cc:1398] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:40.115676 2131 tablet_bootstrap.cc:492] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87: Bootstrap starting.
I20240502 01:58:40.115881 2130 tablet_bootstrap.cc:654] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.116497 2131 tablet_bootstrap.cc:654] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.116657 2132 tablet_bootstrap.cc:492] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683: No bootstrap required, opened a new log
I20240502 01:58:40.116739 2132 ts_tablet_manager.cc:1367] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:40.116957 2130 tablet_bootstrap.cc:492] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072: No bootstrap required, opened a new log
I20240502 01:58:40.117028 2130 ts_tablet_manager.cc:1367] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:40.117200 2132 raft_consensus.cc:348] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.117300 2132 raft_consensus.cc:374] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.117332 2132 raft_consensus.cc:729] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.117435 2131 tablet_bootstrap.cc:492] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87: No bootstrap required, opened a new log
I20240502 01:58:40.117448 2132 consensus_queue.cc:260] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.117465 2130 raft_consensus.cc:348] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:40.117530 2131 ts_tablet_manager.cc:1367] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:40.117545 2130 raft_consensus.cc:374] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.117576 2130 raft_consensus.cc:729] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f073f261d9447b99220aa532c2b1072, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.117723 2132 ts_tablet_manager.cc:1398] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:40.117686 2130 consensus_queue.cc:260] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:40.117929 2130 ts_tablet_manager.cc:1398] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:40.117926 2131 raft_consensus.cc:348] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:40.118016 2131 raft_consensus.cc:374] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.118048 2131 raft_consensus.cc:729] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.118077 2130 tablet_bootstrap.cc:492] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072: Bootstrap starting.
I20240502 01:58:40.118157 2131 consensus_queue.cc:260] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:40.118458 2131 ts_tablet_manager.cc:1398] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.118618 2131 tablet_bootstrap.cc:492] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87: Bootstrap starting.
I20240502 01:58:40.119171 2130 tablet_bootstrap.cc:654] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.119441 2131 tablet_bootstrap.cc:654] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.120893 2130 tablet_bootstrap.cc:492] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072: No bootstrap required, opened a new log
I20240502 01:58:40.120980 2130 ts_tablet_manager.cc:1367] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:40.121352 2131 tablet_bootstrap.cc:492] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87: No bootstrap required, opened a new log
I20240502 01:58:40.121443 2131 ts_tablet_manager.cc:1367] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:40.121404 2130 raft_consensus.cc:348] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.121511 2130 raft_consensus.cc:374] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.121554 2130 raft_consensus.cc:729] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f073f261d9447b99220aa532c2b1072, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.121665 2130 consensus_queue.cc:260] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.121884 2131 raft_consensus.cc:348] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.121949 2130 ts_tablet_manager.cc:1398] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.121965 2131 raft_consensus.cc:374] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.122001 2131 raft_consensus.cc:729] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.122112 2131 consensus_queue.cc:260] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.122363 2131 ts_tablet_manager.cc:1398] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:40.140401 2014 raft_consensus.cc:482] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:40.140527 2014 raft_consensus.cc:504] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.140836 2014 leader_election.cc:288] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:40.141248 1256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a946d0b927414f88c615d54ce5fc7f" candidate_uuid: "3f073f261d9447b99220aa532c2b1072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" is_pre_election: true
I20240502 01:58:40.141391 1256 raft_consensus.cc:2454] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f073f261d9447b99220aa532c2b1072 in term 0.
I20240502 01:58:40.141716 1320 leader_election.cc:302] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, 4d8c8df1c10b41fab826db1b2ef82e87; no voters:
I20240502 01:58:40.141928 2014 raft_consensus.cc:2781] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:40.141990 2014 raft_consensus.cc:482] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:40.142020 2014 raft_consensus.cc:3037] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.142959 2014 raft_consensus.cc:504] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.143230 2014 leader_election.cc:288] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [CANDIDATE]: Term 1 election: Requested vote from peers 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:40.143622 1256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a946d0b927414f88c615d54ce5fc7f" candidate_uuid: "3f073f261d9447b99220aa532c2b1072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d8c8df1c10b41fab826db1b2ef82e87"
I20240502 01:58:40.143728 1256 raft_consensus.cc:3037] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.144501 1256 raft_consensus.cc:2454] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f073f261d9447b99220aa532c2b1072 in term 1.
I20240502 01:58:40.144802 1320 leader_election.cc:302] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, 4d8c8df1c10b41fab826db1b2ef82e87; no voters:
I20240502 01:58:40.144971 2014 raft_consensus.cc:2781] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:40.145161 2014 raft_consensus.cc:686] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [term 1 LEADER]: Becoming Leader. State: Replica: 3f073f261d9447b99220aa532c2b1072, State: Running, Role: LEADER
I20240502 01:58:40.145324 2014 consensus_queue.cc:237] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.146905 974 catalog_manager.cc:5484] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f073f261d9447b99220aa532c2b1072 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "3f073f261d9447b99220aa532c2b1072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } }
I20240502 01:58:40.193729 2065 raft_consensus.cc:482] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:40.193914 2065 raft_consensus.cc:504] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:40.194226 2065 leader_election.cc:288] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:40.194669 1387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "299019dc3d0848ad9324a84e319b9222" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f073f261d9447b99220aa532c2b1072" is_pre_election: true
I20240502 01:58:40.194808 1387 raft_consensus.cc:2454] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 0.
I20240502 01:58:40.195137 1190 leader_election.cc:302] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, 4d8c8df1c10b41fab826db1b2ef82e87; no voters:
I20240502 01:58:40.195325 2065 raft_consensus.cc:2781] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:40.195391 2065 raft_consensus.cc:482] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:40.195438 2065 raft_consensus.cc:3037] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.196393 2065 raft_consensus.cc:504] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:40.196661 2065 leader_election.cc:288] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Requested vote from peers 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:40.197022 1387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "299019dc3d0848ad9324a84e319b9222" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f073f261d9447b99220aa532c2b1072"
I20240502 01:58:40.197125 1387 raft_consensus.cc:3037] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.197873 1387 raft_consensus.cc:2454] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 1.
I20240502 01:58:40.198145 1190 leader_election.cc:302] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, 4d8c8df1c10b41fab826db1b2ef82e87; no voters:
I20240502 01:58:40.198295 2065 raft_consensus.cc:2781] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:40.198412 2065 raft_consensus.cc:686] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Becoming Leader. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Running, Role: LEADER
I20240502 01:58:40.198556 2065 consensus_queue.cc:237] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:40.200181 974 catalog_manager.cc:5484] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } }
I20240502 01:58:40.308847 2065 raft_consensus.cc:482] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:40.308972 2065 raft_consensus.cc:504] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.309324 2065 leader_election.cc:288] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:40.309808 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa93c869b01b4af69678e378df7db684" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a571e02bbe774f45b80bb4383642c683" is_pre_election: true
I20240502 01:58:40.309939 1125 raft_consensus.cc:2454] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 0.
I20240502 01:58:40.310238 1190 leader_election.cc:302] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d8c8df1c10b41fab826db1b2ef82e87, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:40.310456 2065 raft_consensus.cc:2781] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:40.310523 2065 raft_consensus.cc:482] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:40.310554 2065 raft_consensus.cc:3037] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.311390 2065 raft_consensus.cc:504] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.311664 2065 leader_election.cc:288] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Requested vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:40.312192 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa93c869b01b4af69678e378df7db684" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a571e02bbe774f45b80bb4383642c683"
I20240502 01:58:40.312302 1125 raft_consensus.cc:3037] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.313064 1125 raft_consensus.cc:2454] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 1.
I20240502 01:58:40.313387 1190 leader_election.cc:302] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d8c8df1c10b41fab826db1b2ef82e87, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:40.313544 2065 raft_consensus.cc:2781] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:40.313642 2065 raft_consensus.cc:686] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Becoming Leader. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Running, Role: LEADER
I20240502 01:58:40.313779 2065 consensus_queue.cc:237] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.315486 974 catalog_manager.cc:5484] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } }
W20240502 01:58:40.456486 2143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:40.456732 2143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:40.459753 2143 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 01:58:40.530894 2014 consensus_queue.cc:1035] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:40.532367 1387 raft_consensus.cc:1261] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:40.532989 2064 consensus_queue.cc:1035] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:40.535504 1125 raft_consensus.cc:1261] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:40.535985 2064 consensus_queue.cc:1035] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 76.3751 ms
time per row: 0.0381875 ms
I20240502 01:58:40.576786 982 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34484:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:40.582526 1103 tablet_service.cc:1467] Processing CreateTablet for tablet 90948f77e44241e1883249fbcdcf1515 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f35b0e8f48d64f5884fb03539ed2621e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.582666 1367 tablet_service.cc:1467] Processing CreateTablet for tablet 90948f77e44241e1883249fbcdcf1515 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f35b0e8f48d64f5884fb03539ed2621e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.582850 1103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90948f77e44241e1883249fbcdcf1515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.582902 1367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90948f77e44241e1883249fbcdcf1515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.582928 1234 tablet_service.cc:1467] Processing CreateTablet for tablet dd52517cbc5b43b798af83233b3759b9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f35b0e8f48d64f5884fb03539ed2621e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.583181 1234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd52517cbc5b43b798af83233b3759b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.585093 1366 tablet_service.cc:1467] Processing CreateTablet for tablet a6da3730570e48d2aba7457e9d41dfcc (DEFAULT_TABLE table=rebalance_test_table_3 [id=f35b0e8f48d64f5884fb03539ed2621e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.585182 2131 tablet_bootstrap.cc:492] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87: Bootstrap starting.
I20240502 01:58:40.585337 1366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6da3730570e48d2aba7457e9d41dfcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.585973 2131 tablet_bootstrap.cc:654] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.585104 2130 tablet_bootstrap.cc:492] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072: Bootstrap starting.
I20240502 01:58:40.586901 2132 tablet_bootstrap.cc:492] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683: Bootstrap starting.
I20240502 01:58:40.587334 2130 tablet_bootstrap.cc:654] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.587770 2132 tablet_bootstrap.cc:654] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.588632 1104 tablet_service.cc:1467] Processing CreateTablet for tablet dd52517cbc5b43b798af83233b3759b9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f35b0e8f48d64f5884fb03539ed2621e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.588881 1104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd52517cbc5b43b798af83233b3759b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.588896 2131 tablet_bootstrap.cc:492] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87: No bootstrap required, opened a new log
I20240502 01:58:40.588979 2131 ts_tablet_manager.cc:1367] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:58:40.589417 2131 raft_consensus.cc:348] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.589521 2131 raft_consensus.cc:374] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.589551 2131 raft_consensus.cc:729] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.589682 2131 consensus_queue.cc:260] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.589970 2131 ts_tablet_manager.cc:1398] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:40.588680 2132 tablet_bootstrap.cc:492] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683: No bootstrap required, opened a new log
I20240502 01:58:40.590895 2132 ts_tablet_manager.cc:1367] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:40.591291 2132 raft_consensus.cc:348] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:40.591387 2132 raft_consensus.cc:374] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.591418 2132 raft_consensus.cc:729] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.591538 2132 consensus_queue.cc:260] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:40.591821 2132 ts_tablet_manager.cc:1398] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.591975 2132 tablet_bootstrap.cc:492] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683: Bootstrap starting.
I20240502 01:58:40.592749 2130 tablet_bootstrap.cc:492] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072: No bootstrap required, opened a new log
I20240502 01:58:40.592811 2132 tablet_bootstrap.cc:654] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.592825 2130 ts_tablet_manager.cc:1367] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 01:58:40.593278 2130 raft_consensus.cc:348] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:40.593379 2130 raft_consensus.cc:374] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.593417 2130 raft_consensus.cc:729] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f073f261d9447b99220aa532c2b1072, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.593575 2130 consensus_queue.cc:260] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:40.593900 2130 ts_tablet_manager.cc:1398] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:40.594045 2130 tablet_bootstrap.cc:492] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072: Bootstrap starting.
I20240502 01:58:40.594142 2132 tablet_bootstrap.cc:492] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683: No bootstrap required, opened a new log
I20240502 01:58:40.594233 2132 ts_tablet_manager.cc:1367] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:40.594702 1103 tablet_service.cc:1467] Processing CreateTablet for tablet a6da3730570e48d2aba7457e9d41dfcc (DEFAULT_TABLE table=rebalance_test_table_3 [id=f35b0e8f48d64f5884fb03539ed2621e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:40.594729 2132 raft_consensus.cc:348] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.594816 2132 raft_consensus.cc:374] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.594848 2132 raft_consensus.cc:729] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.594940 1103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6da3730570e48d2aba7457e9d41dfcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:40.595198 2130 tablet_bootstrap.cc:654] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.594995 2132 consensus_queue.cc:260] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.595669 2132 ts_tablet_manager.cc:1398] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:40.597328 2132 tablet_bootstrap.cc:492] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683: Bootstrap starting.
I20240502 01:58:40.597404 2130 tablet_bootstrap.cc:492] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072: No bootstrap required, opened a new log
I20240502 01:58:40.597491 2130 ts_tablet_manager.cc:1367] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:58:40.597873 2130 raft_consensus.cc:348] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:40.597986 2130 raft_consensus.cc:374] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.598021 2130 raft_consensus.cc:729] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f073f261d9447b99220aa532c2b1072, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.598134 2132 tablet_bootstrap.cc:654] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:40.598152 2130 consensus_queue.cc:260] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:40.598456 2130 ts_tablet_manager.cc:1398] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:40.599339 2132 tablet_bootstrap.cc:492] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683: No bootstrap required, opened a new log
I20240502 01:58:40.599416 2132 ts_tablet_manager.cc:1367] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:40.599753 2132 raft_consensus.cc:348] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:40.599851 2132 raft_consensus.cc:374] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:40.599901 2132 raft_consensus.cc:729] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Initialized, Role: FOLLOWER
I20240502 01:58:40.600019 2132 consensus_queue.cc:260] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:40.600267 2132 ts_tablet_manager.cc:1398] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:40.766528 2174 raft_consensus.cc:482] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:40.766690 2174 raft_consensus.cc:504] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.767082 2174 leader_election.cc:288] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:40.767642 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd52517cbc5b43b798af83233b3759b9" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a571e02bbe774f45b80bb4383642c683" is_pre_election: true
I20240502 01:58:40.767772 1125 raft_consensus.cc:2454] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 0.
I20240502 01:58:40.768144 1190 leader_election.cc:302] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d8c8df1c10b41fab826db1b2ef82e87, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:40.768396 2174 raft_consensus.cc:2781] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:40.768450 2174 raft_consensus.cc:482] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:40.768479 2174 raft_consensus.cc:3037] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.769368 2174 raft_consensus.cc:504] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.769636 2174 leader_election.cc:288] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Requested vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:40.770031 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd52517cbc5b43b798af83233b3759b9" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a571e02bbe774f45b80bb4383642c683"
I20240502 01:58:40.770133 1125 raft_consensus.cc:3037] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.770943 1125 raft_consensus.cc:2454] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 1.
I20240502 01:58:40.771226 1190 leader_election.cc:302] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d8c8df1c10b41fab826db1b2ef82e87, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:40.771376 2174 raft_consensus.cc:2781] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:40.771456 2174 raft_consensus.cc:686] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Becoming Leader. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Running, Role: LEADER
I20240502 01:58:40.771600 2174 consensus_queue.cc:237] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } }
I20240502 01:58:40.773445 978 catalog_manager.cc:5484] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } }
I20240502 01:58:40.814019 2014 raft_consensus.cc:482] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:40.814182 2014 raft_consensus.cc:504] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:40.814700 2014 leader_election.cc:288] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:40.818568 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6da3730570e48d2aba7457e9d41dfcc" candidate_uuid: "3f073f261d9447b99220aa532c2b1072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a571e02bbe774f45b80bb4383642c683" is_pre_election: true
I20240502 01:58:40.818709 1125 raft_consensus.cc:2454] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f073f261d9447b99220aa532c2b1072 in term 0.
I20240502 01:58:40.819016 1321 leader_election.cc:302] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:40.819208 2014 raft_consensus.cc:2781] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:40.819273 2014 raft_consensus.cc:482] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:40.819320 2014 raft_consensus.cc:3037] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.820161 2014 raft_consensus.cc:504] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:40.820473 2014 leader_election.cc:288] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [CANDIDATE]: Term 1 election: Requested vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:40.820849 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6da3730570e48d2aba7457e9d41dfcc" candidate_uuid: "3f073f261d9447b99220aa532c2b1072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a571e02bbe774f45b80bb4383642c683"
I20240502 01:58:40.820979 1125 raft_consensus.cc:3037] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.821703 1125 raft_consensus.cc:2454] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f073f261d9447b99220aa532c2b1072 in term 1.
I20240502 01:58:40.821983 1321 leader_election.cc:302] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:40.822144 2014 raft_consensus.cc:2781] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:40.822233 2014 raft_consensus.cc:686] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [term 1 LEADER]: Becoming Leader. State: Replica: 3f073f261d9447b99220aa532c2b1072, State: Running, Role: LEADER
I20240502 01:58:40.822372 2014 consensus_queue.cc:237] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:40.823940 978 catalog_manager.cc:5484] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f073f261d9447b99220aa532c2b1072 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "3f073f261d9447b99220aa532c2b1072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } }
I20240502 01:58:40.979873 2056 raft_consensus.cc:482] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:40.980028 2056 raft_consensus.cc:504] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:40.980365 2056 leader_election.cc:288] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:40.980918 1387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90948f77e44241e1883249fbcdcf1515" candidate_uuid: "a571e02bbe774f45b80bb4383642c683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f073f261d9447b99220aa532c2b1072" is_pre_election: true
I20240502 01:58:40.981079 1387 raft_consensus.cc:2454] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a571e02bbe774f45b80bb4383642c683 in term 0.
I20240502 01:58:40.981426 1059 leader_election.cc:302] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:40.981654 2056 raft_consensus.cc:2781] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:40.981724 2056 raft_consensus.cc:482] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:40.981755 2056 raft_consensus.cc:3037] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.982705 2056 raft_consensus.cc:504] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:40.982988 2056 leader_election.cc:288] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 election: Requested vote from peers 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:40.983414 1387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90948f77e44241e1883249fbcdcf1515" candidate_uuid: "a571e02bbe774f45b80bb4383642c683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f073f261d9447b99220aa532c2b1072"
I20240502 01:58:40.983543 1387 raft_consensus.cc:3037] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:40.984329 1387 raft_consensus.cc:2454] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a571e02bbe774f45b80bb4383642c683 in term 1.
I20240502 01:58:40.984619 1059 leader_election.cc:302] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:40.984800 2056 raft_consensus.cc:2781] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:40.984949 2056 raft_consensus.cc:686] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Becoming Leader. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Running, Role: LEADER
I20240502 01:58:40.985078 2056 consensus_queue.cc:237] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:40.986750 978 catalog_manager.cc:5484] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 reported cstate change: term changed from 0 to 1, leader changed from <none> to a571e02bbe774f45b80bb4383642c683 (127.27.111.193). New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } }
W20240502 01:58:41.150050 2188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:41.150303 2188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:41.153419 2188 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 01:58:41.233983 1387 raft_consensus.cc:1261] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:41.234558 2056 consensus_queue.cc:1035] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:41.236550 1125 raft_consensus.cc:1261] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Refusing update from remote peer 3f073f261d9447b99220aa532c2b1072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:41.238040 1125 raft_consensus.cc:1261] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:41.238240 2014 consensus_queue.cc:1035] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:41.238461 2174 consensus_queue.cc:1035] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 91.2397 ms
time per row: 0.0456199 ms
I20240502 01:58:41.287624 982 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34484:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:58:41.293367 1366 tablet_service.cc:1467] Processing CreateTablet for tablet a59a5084412f4082bdd35e7c546e71b4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ebb17b8513b14eb0b4a24e8de3c351bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.293377 1234 tablet_service.cc:1467] Processing CreateTablet for tablet a59a5084412f4082bdd35e7c546e71b4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ebb17b8513b14eb0b4a24e8de3c351bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.293677 1234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59a5084412f4082bdd35e7c546e71b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.293684 1366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59a5084412f4082bdd35e7c546e71b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.293962 1103 tablet_service.cc:1467] Processing CreateTablet for tablet 690f9ec1b4c34a919a9a09ae7c618ef3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ebb17b8513b14eb0b4a24e8de3c351bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.294215 1103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690f9ec1b4c34a919a9a09ae7c618ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.296013 2241 tablet_bootstrap.cc:492] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87: Bootstrap starting.
I20240502 01:58:41.296504 1366 tablet_service.cc:1467] Processing CreateTablet for tablet 5b90b275dd77400f9aaa1ffc0ea4be18 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ebb17b8513b14eb0b4a24e8de3c351bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.296751 1366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b90b275dd77400f9aaa1ffc0ea4be18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.296734 1236 tablet_service.cc:1467] Processing CreateTablet for tablet 690f9ec1b4c34a919a9a09ae7c618ef3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ebb17b8513b14eb0b4a24e8de3c351bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.296931 2241 tablet_bootstrap.cc:654] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.296958 1236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690f9ec1b4c34a919a9a09ae7c618ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.297469 2242 tablet_bootstrap.cc:492] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683: Bootstrap starting.
I20240502 01:58:41.298347 2242 tablet_bootstrap.cc:654] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.298763 2240 tablet_bootstrap.cc:492] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072: Bootstrap starting.
I20240502 01:58:41.299149 1234 tablet_service.cc:1467] Processing CreateTablet for tablet 5b90b275dd77400f9aaa1ffc0ea4be18 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ebb17b8513b14eb0b4a24e8de3c351bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:58:41.299388 1234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b90b275dd77400f9aaa1ffc0ea4be18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:41.299589 2240 tablet_bootstrap.cc:654] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.300135 2242 tablet_bootstrap.cc:492] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683: No bootstrap required, opened a new log
I20240502 01:58:41.300225 2242 ts_tablet_manager.cc:1367] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:41.300585 2241 tablet_bootstrap.cc:492] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87: No bootstrap required, opened a new log
I20240502 01:58:41.300663 2241 ts_tablet_manager.cc:1367] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:58:41.300719 2242 raft_consensus.cc:348] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:41.300837 2242 raft_consensus.cc:374] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.300886 2242 raft_consensus.cc:729] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.301023 2242 consensus_queue.cc:260] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:41.301079 2241 raft_consensus.cc:348] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:41.301159 2241 raft_consensus.cc:374] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.301192 2241 raft_consensus.cc:729] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.301290 2242 ts_tablet_manager.cc:1398] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.301307 2241 consensus_queue.cc:260] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:41.301604 2241 ts_tablet_manager.cc:1398] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.301733 2241 tablet_bootstrap.cc:492] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87: Bootstrap starting.
I20240502 01:58:41.301779 2240 tablet_bootstrap.cc:492] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072: No bootstrap required, opened a new log
I20240502 01:58:41.301852 2240 ts_tablet_manager.cc:1367] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:41.302273 2240 raft_consensus.cc:348] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:41.302618 2241 tablet_bootstrap.cc:654] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.302384 2240 raft_consensus.cc:374] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.302753 2240 raft_consensus.cc:729] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f073f261d9447b99220aa532c2b1072, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.302870 2240 consensus_queue.cc:260] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:41.303136 2240 ts_tablet_manager.cc:1398] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.303257 2240 tablet_bootstrap.cc:492] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 3f073f261d9447b99220aa532c2b1072: Bootstrap starting.
I20240502 01:58:41.303993 2241 tablet_bootstrap.cc:492] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87: No bootstrap required, opened a new log
I20240502 01:58:41.304073 2240 tablet_bootstrap.cc:654] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 3f073f261d9447b99220aa532c2b1072: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.304136 2241 ts_tablet_manager.cc:1367] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:41.304569 2241 raft_consensus.cc:348] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:41.304657 2241 raft_consensus.cc:374] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.304687 2241 raft_consensus.cc:729] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.304821 2241 consensus_queue.cc:260] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:41.305205 2241 ts_tablet_manager.cc:1398] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.305377 2241 tablet_bootstrap.cc:492] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87: Bootstrap starting.
I20240502 01:58:41.306459 2241 tablet_bootstrap.cc:654] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87: Neither blocks nor log segments found. Creating new log.
I20240502 01:58:41.306896 2240 tablet_bootstrap.cc:492] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 3f073f261d9447b99220aa532c2b1072: No bootstrap required, opened a new log
I20240502 01:58:41.306982 2240 ts_tablet_manager.cc:1367] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 3f073f261d9447b99220aa532c2b1072: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:41.307444 2240 raft_consensus.cc:348] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:41.307547 2240 raft_consensus.cc:374] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.307603 2240 raft_consensus.cc:729] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f073f261d9447b99220aa532c2b1072, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.307726 2240 consensus_queue.cc:260] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 3f073f261d9447b99220aa532c2b1072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:41.307988 2240 ts_tablet_manager.cc:1398] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 3f073f261d9447b99220aa532c2b1072: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:41.308775 2241 tablet_bootstrap.cc:492] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87: No bootstrap required, opened a new log
I20240502 01:58:41.308892 2241 ts_tablet_manager.cc:1367] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:58:41.309326 2241 raft_consensus.cc:348] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:41.309518 2241 raft_consensus.cc:374] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:58:41.309573 2241 raft_consensus.cc:729] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Initialized, Role: FOLLOWER
I20240502 01:58:41.309733 2241 consensus_queue.cc:260] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:41.310060 2241 ts_tablet_manager.cc:1398] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:58:41.354074 2056 raft_consensus.cc:482] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:41.354218 2056 raft_consensus.cc:504] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:41.354637 2056 leader_election.cc:288] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07c427880bd54215847186eab221fce5 (127.27.111.197:39309)
W20240502 01:58:41.356181 1061 leader_election.cc:334] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 07c427880bd54215847186eab221fce5 (127.27.111.197:39309): Network error: Client connection negotiation failed: client connection to 127.27.111.197:39309: connect: Connection refused (error 111)
I20240502 01:58:41.356256 1061 leader_election.cc:302] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a571e02bbe774f45b80bb4383642c683; no voters: 07c427880bd54215847186eab221fce5
I20240502 01:58:41.356386 2056 raft_consensus.cc:2726] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:58:41.357551 2056 raft_consensus.cc:482] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:41.357622 2056 raft_consensus.cc:504] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:41.357954 2056 leader_election.cc:288] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:41.361354 1256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690f9ec1b4c34a919a9a09ae7c618ef3" candidate_uuid: "a571e02bbe774f45b80bb4383642c683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" is_pre_election: true
I20240502 01:58:41.361506 1256 raft_consensus.cc:2454] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a571e02bbe774f45b80bb4383642c683 in term 0.
I20240502 01:58:41.361824 1058 leader_election.cc:302] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d8c8df1c10b41fab826db1b2ef82e87, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:41.362008 2056 raft_consensus.cc:2781] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:41.362077 2056 raft_consensus.cc:482] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:41.362107 2056 raft_consensus.cc:3037] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.362943 2056 raft_consensus.cc:504] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:41.363200 2056 leader_election.cc:288] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 election: Requested vote from peers 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:41.363555 1256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690f9ec1b4c34a919a9a09ae7c618ef3" candidate_uuid: "a571e02bbe774f45b80bb4383642c683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d8c8df1c10b41fab826db1b2ef82e87"
I20240502 01:58:41.363688 1256 raft_consensus.cc:3037] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.364444 1256 raft_consensus.cc:2454] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a571e02bbe774f45b80bb4383642c683 in term 1.
I20240502 01:58:41.364742 1058 leader_election.cc:302] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4d8c8df1c10b41fab826db1b2ef82e87, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:41.364902 2056 raft_consensus.cc:2781] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:41.364988 2056 raft_consensus.cc:686] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Becoming Leader. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Running, Role: LEADER
I20240502 01:58:41.365113 2056 consensus_queue.cc:237] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } }
I20240502 01:58:41.366816 978 catalog_manager.cc:5484] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 reported cstate change: term changed from 0 to 1, leader changed from <none> to a571e02bbe774f45b80bb4383642c683 (127.27.111.193). New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } }
I20240502 01:58:41.369077 2174 raft_consensus.cc:482] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:41.369165 2174 raft_consensus.cc:504] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:41.369450 2174 leader_election.cc:288] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:41.369853 1387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b90b275dd77400f9aaa1ffc0ea4be18" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f073f261d9447b99220aa532c2b1072" is_pre_election: true
I20240502 01:58:41.369977 1387 raft_consensus.cc:2454] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 0.
I20240502 01:58:41.370254 1190 leader_election.cc:302] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, 4d8c8df1c10b41fab826db1b2ef82e87; no voters:
I20240502 01:58:41.370443 2174 raft_consensus.cc:2781] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:41.370501 2174 raft_consensus.cc:482] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:41.370528 2174 raft_consensus.cc:3037] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.371277 2174 raft_consensus.cc:504] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:41.371513 2174 leader_election.cc:288] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Requested vote from peers 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:41.371865 1387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b90b275dd77400f9aaa1ffc0ea4be18" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f073f261d9447b99220aa532c2b1072"
I20240502 01:58:41.371965 1387 raft_consensus.cc:3037] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.372684 1387 raft_consensus.cc:2454] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 1.
I20240502 01:58:41.372952 1190 leader_election.cc:302] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, 4d8c8df1c10b41fab826db1b2ef82e87; no voters:
I20240502 01:58:41.373102 2174 raft_consensus.cc:2781] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:41.373181 2174 raft_consensus.cc:686] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Becoming Leader. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Running, Role: LEADER
I20240502 01:58:41.373304 2174 consensus_queue.cc:237] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:41.375054 974 catalog_manager.cc:5484] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } }
I20240502 01:58:41.486811 2174 raft_consensus.cc:482] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:41.486953 2174 raft_consensus.cc:504] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:41.487310 2174 leader_election.cc:288] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:41.487779 1387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59a5084412f4082bdd35e7c546e71b4" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f073f261d9447b99220aa532c2b1072" is_pre_election: true
I20240502 01:58:41.487922 1387 raft_consensus.cc:2454] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 0.
I20240502 01:58:41.488229 1190 leader_election.cc:302] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, 4d8c8df1c10b41fab826db1b2ef82e87; no voters:
I20240502 01:58:41.488405 2174 raft_consensus.cc:2781] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:58:41.488471 2174 raft_consensus.cc:482] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:41.488502 2174 raft_consensus.cc:3037] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.489352 2174 raft_consensus.cc:504] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:41.489602 2174 leader_election.cc:288] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Requested vote from peers 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:41.490006 1387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59a5084412f4082bdd35e7c546e71b4" candidate_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f073f261d9447b99220aa532c2b1072"
I20240502 01:58:41.490150 1387 raft_consensus.cc:3037] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:58:41.490974 1387 raft_consensus.cc:2454] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d8c8df1c10b41fab826db1b2ef82e87 in term 1.
I20240502 01:58:41.491257 1190 leader_election.cc:302] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 3f073f261d9447b99220aa532c2b1072, 4d8c8df1c10b41fab826db1b2ef82e87; no voters:
I20240502 01:58:41.491405 2174 raft_consensus.cc:2781] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:58:41.491490 2174 raft_consensus.cc:686] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Becoming Leader. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Running, Role: LEADER
I20240502 01:58:41.491636 2174 consensus_queue.cc:237] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } }
I20240502 01:58:41.493315 974 catalog_manager.cc:5484] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } }
W20240502 01:58:41.643817 2254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:41.644078 2254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:41.647136 2254 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 01:58:41.730185 1387 raft_consensus.cc:1261] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:41.730736 2174 consensus_queue.cc:1035] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:41.730952 1387 raft_consensus.cc:1261] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:41.731439 2174 consensus_queue.cc:1035] T 5b90b275dd77400f9aaa1ffc0ea4be18 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:41.732398 1256 raft_consensus.cc:1261] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:58:41.732779 2056 consensus_queue.cc:1035] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 97.3479 ms
time per row: 0.0486739 ms
I20240502 01:58:41.785763 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.196:43819
--local_ip_for_outbound_sockets=127.27.111.196
--tserver_master_addrs=127.27.111.254:38251
--webserver_port=39957
--webserver_interface=127.27.111.196
--builtin_ntp_servers=127.27.111.212:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:58:41.894198 2285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:41.894418 2285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:41.894460 2285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:41.897506 2285 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:41.897564 2285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:41.897645 2285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.196
I20240502 01:58:41.901273 2285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.196:43819
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.111.196
--webserver_port=39957
--tserver_master_addrs=127.27.111.254:38251
--never_fsync=true
--heap_profile_path=/tmp/kudu.2285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.196
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:41.902197 2285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:41.903329 2285 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:58:41.912310 2285 server_base.cc:1034] running on GCE node
W20240502 01:58:41.912811 2293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:41.913429 2290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:41.914235 2291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:41.914796 2285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:41.915405 2285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:41.916584 2285 hybrid_clock.cc:648] HybridClock initialized: now 1714615121916569 us; error 32 us; skew 500 ppm
I20240502 01:58:41.918509 2285 webserver.cc:460] Webserver started at http://127.27.111.196:39957/ using document root <none> and password file <none>
I20240502 01:58:41.919070 2285 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:41.919148 2285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:41.922384 2285 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 01:58:41.924177 2299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:41.925132 2285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:41.925246 2285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "09401a31cfd046a78a67a2d51e290224"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:41.925683 2285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:41.938225 2285 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:41.938871 2285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:41.939026 2285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:41.939548 2285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:41.940449 2285 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:41.940500 2285 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:41.940563 2285 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:41.940591 2285 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:41.949702 2285 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.196:43819
I20240502 01:58:41.949867 2412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.196:43819 every 8 connection(s)
I20240502 01:58:41.951634 2285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:58:41.961305 2413 heartbeater.cc:347] Connected to a master server at 127.27.111.254:38251
I20240502 01:58:41.961555 2413 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:41.961675 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 2285
I20240502 01:58:41.962133 2413 heartbeater.cc:511] Master 127.27.111.254:38251 requested a full tablet report, sending...
I20240502 01:58:41.962123 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.197:39309
--local_ip_for_outbound_sockets=127.27.111.197
--tserver_master_addrs=127.27.111.254:38251
--webserver_port=41999
--webserver_interface=127.27.111.197
--builtin_ntp_servers=127.27.111.212:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:58:41.963380 982 ts_manager.cc:194] Re-registered known tserver with Master: 09401a31cfd046a78a67a2d51e290224 (127.27.111.196:43819)
I20240502 01:58:41.964118 982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.196:60161
W20240502 01:58:42.069253 2416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:42.069445 2416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:42.069483 2416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:42.072196 2416 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:42.072254 2416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:42.072316 2416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.197
I20240502 01:58:42.075692 2416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.197:39309
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.111.197
--webserver_port=41999
--tserver_master_addrs=127.27.111.254:38251
--never_fsync=true
--heap_profile_path=/tmp/kudu.2416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.197
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:42.076506 2416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:42.077639 2416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:42.086117 2422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:42.086581 2421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:42.087397 2424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:42.088150 2416 server_base.cc:1034] running on GCE node
I20240502 01:58:42.088479 2416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:42.088994 2416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:42.090139 2416 hybrid_clock.cc:648] HybridClock initialized: now 1714615122090130 us; error 28 us; skew 500 ppm
I20240502 01:58:42.091799 2416 webserver.cc:460] Webserver started at http://127.27.111.197:41999/ using document root <none> and password file <none>
I20240502 01:58:42.092305 2416 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:42.092381 2416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:42.095650 2416 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:58:42.097595 2430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:42.098587 2416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:42.098671 2416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "07c427880bd54215847186eab221fce5"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:42.099058 2416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:42.132692 2416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:42.133366 2416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:42.133505 2416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:42.134049 2416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:42.135190 2437 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240502 01:58:42.138368 2416 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 01:58:42.138459 2416 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:42.138535 2416 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 01:58:42.139959 2416 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 01:58:42.140003 2416 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240502 01:58:42.140254 2437 tablet_bootstrap.cc:492] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5: Bootstrap starting.
I20240502 01:58:42.151018 2416 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.197:39309
I20240502 01:58:42.151055 2544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.197:39309 every 8 connection(s)
I20240502 01:58:42.152065 2416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:58:42.154256 2437 log.cc:824] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:42.156332 2437 tablet_bootstrap.cc:492] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:58:42.156715 2437 tablet_bootstrap.cc:492] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5: Bootstrap complete.
I20240502 01:58:42.157683 2437 ts_tablet_manager.cc:1367] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20240502 01:58:42.158589 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 2416
I20240502 01:58:42.159015 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.198:45287
--local_ip_for_outbound_sockets=127.27.111.198
--tserver_master_addrs=127.27.111.254:38251
--webserver_port=39591
--webserver_interface=127.27.111.198
--builtin_ntp_servers=127.27.111.212:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:58:42.160472 2545 heartbeater.cc:347] Connected to a master server at 127.27.111.254:38251
I20240502 01:58:42.160677 2545 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:42.161118 2545 heartbeater.cc:511] Master 127.27.111.254:38251 requested a full tablet report, sending...
I20240502 01:58:42.161252 2437 raft_consensus.cc:348] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:42.161746 2437 raft_consensus.cc:729] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c427880bd54215847186eab221fce5, State: Initialized, Role: FOLLOWER
I20240502 01:58:42.162158 2437 consensus_queue.cc:260] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:42.162961 2437 ts_tablet_manager.cc:1398] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20240502 01:58:42.164122 982 ts_manager.cc:194] Re-registered known tserver with Master: 07c427880bd54215847186eab221fce5 (127.27.111.197:39309)
I20240502 01:58:42.164858 982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.197:60409
I20240502 01:58:42.166735 2545 heartbeater.cc:503] Master 127.27.111.254:38251 was elected leader, sending a full tablet report...
W20240502 01:58:42.273150 2549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:42.273365 2549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:42.273434 2549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:42.276741 2549 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:42.276821 2549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:42.276944 2549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.198
I20240502 01:58:42.280462 2549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.198:45287
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.111.198
--webserver_port=39591
--tserver_master_addrs=127.27.111.254:38251
--never_fsync=true
--heap_profile_path=/tmp/kudu.2549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.198
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:42.281327 2549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:42.282413 2549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:42.291033 2555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:42.291712 2556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:42.292483 2558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:42.293148 2549 server_base.cc:1034] running on GCE node
I20240502 01:58:42.293511 2549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:42.294076 2549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:42.295234 2549 hybrid_clock.cc:648] HybridClock initialized: now 1714615122295223 us; error 27 us; skew 500 ppm
I20240502 01:58:42.297001 2549 webserver.cc:460] Webserver started at http://127.27.111.198:39591/ using document root <none> and password file <none>
I20240502 01:58:42.297570 2549 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:42.297655 2549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:42.301275 2549 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:42.303381 2564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:42.304454 2549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:42.304561 2549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "72de8d04b745434f923594c920aff90c"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:42.305011 2549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:42.328368 2549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:42.329291 2549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:42.329622 2549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:42.330384 2549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:42.331686 2549 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:42.331782 2549 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:42.331877 2549 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:42.331951 2549 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:42.341485 2549 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.198:45287
I20240502 01:58:42.341528 2677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.198:45287 every 8 connection(s)
I20240502 01:58:42.342727 2549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:58:42.348752 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 2549
I20240502 01:58:42.349218 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.199:33361
--local_ip_for_outbound_sockets=127.27.111.199
--tserver_master_addrs=127.27.111.254:38251
--webserver_port=43063
--webserver_interface=127.27.111.199
--builtin_ntp_servers=127.27.111.212:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:58:42.351629 2678 heartbeater.cc:347] Connected to a master server at 127.27.111.254:38251
I20240502 01:58:42.351866 2678 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:42.352412 2678 heartbeater.cc:511] Master 127.27.111.254:38251 requested a full tablet report, sending...
I20240502 01:58:42.353473 982 ts_manager.cc:194] Re-registered known tserver with Master: 72de8d04b745434f923594c920aff90c (127.27.111.198:45287)
I20240502 01:58:42.354040 982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.198:36119
W20240502 01:58:42.403030 2546 tablet.cc:2369] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:58:42.459084 2681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:58:42.459309 2681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:58:42.459352 2681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:58:42.462314 2681 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:58:42.462373 2681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:58:42.462492 2681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.199
I20240502 01:58:42.465994 2681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.199:33361
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.111.199
--webserver_port=43063
--tserver_master_addrs=127.27.111.254:38251
--never_fsync=true
--heap_profile_path=/tmp/kudu.2681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.199
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:58:42.466889 2681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:58:42.467963 2681 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:58:42.476320 2687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:42.476864 2686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:58:42.477458 2689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:58:42.478379 2681 server_base.cc:1034] running on GCE node
I20240502 01:58:42.478796 2681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:58:42.479346 2681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:58:42.480492 2681 hybrid_clock.cc:648] HybridClock initialized: now 1714615122480483 us; error 26 us; skew 500 ppm
I20240502 01:58:42.482381 2681 webserver.cc:460] Webserver started at http://127.27.111.199:43063/ using document root <none> and password file <none>
I20240502 01:58:42.482959 2681 fs_manager.cc:359] Metadata directory not provided
I20240502 01:58:42.483047 2681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:58:42.486990 2681 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:42.489002 2695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:42.490123 2681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:58:42.490217 2681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "37279883fafe432f81fe7b8224dbe5fb"
format_stamp: "Formatted at 2024-05-02 01:58:34 on dist-test-slave-0pw1"
I20240502 01:58:42.490669 2681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:58:42.501718 2681 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:58:42.502341 2681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:58:42.502527 2681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:58:42.503048 2681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:58:42.503959 2681 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:58:42.504009 2681 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:42.504067 2681 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:58:42.504093 2681 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:58:42.512964 2681 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.199:33361
I20240502 01:58:42.513010 2808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.199:33361 every 8 connection(s)
I20240502 01:58:42.514137 2681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:58:42.515576 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 2681
I20240502 01:58:42.515750 28095 test_util.cc:268] Using random seed: -246556890
I20240502 01:58:42.523623 2809 heartbeater.cc:347] Connected to a master server at 127.27.111.254:38251
I20240502 01:58:42.523846 2809 heartbeater.cc:464] Registering TS with master...
I20240502 01:58:42.524345 2809 heartbeater.cc:511] Master 127.27.111.254:38251 requested a full tablet report, sending...
I20240502 01:58:42.525259 982 ts_manager.cc:194] Re-registered known tserver with Master: 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199:33361)
I20240502 01:58:42.525945 982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.199:40345
I20240502 01:58:42.526476 28095 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 01:58:42.526692 28095 test_util.cc:268] Using random seed: -246545949
I20240502 01:58:42.536168 28095 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 01:58:42.536332 28095 test_util.cc:268] Using random seed: -246536308
I20240502 01:58:42.545549 28095 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 01:58:42.545711 28095 test_util.cc:268] Using random seed: -246526931
I20240502 01:58:42.557143 28095 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 01:58:42.557310 28095 test_util.cc:268] Using random seed: -246515333
I20240502 01:58:42.572476 28095 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240502 01:58:42.966375 2413 heartbeater.cc:503] Master 127.27.111.254:38251 was elected leader, sending a full tablet report...
I20240502 01:58:42.999226 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:43.019342 2479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:43.025097 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:43.025718 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:43.026079 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:43.037911 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:43.046284 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:43.262979 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:43.268077 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:43.277264 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:43.291574 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:43.292598 2479 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:43.302580 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:43.338724 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:43.356542 2678 heartbeater.cc:503] Master 127.27.111.254:38251 was elected leader, sending a full tablet report...
I20240502 01:58:43.435825 1125 consensus_queue.cc:237] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:43.439080 1255 raft_consensus.cc:1261] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 01:58:43.443395 2056 consensus_queue.cc:1035] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240502 01:58:43.444032 1059 consensus_peers.cc:485] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 -> Peer 09401a31cfd046a78a67a2d51e290224 (127.27.111.196:43819): Couldn't send request to peer 09401a31cfd046a78a67a2d51e290224. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 690f9ec1b4c34a919a9a09ae7c618ef3. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.448824 2056 raft_consensus.cc:2932] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:43.451195 1256 raft_consensus.cc:2932] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:43.451669 982 catalog_manager.cc:5484] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 reported cstate change: config changed from index -1 to 28, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:43.510847 1386 consensus_queue.cc:237] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 1.45, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:43.512511 1124 raft_consensus.cc:1261] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Refusing update from remote peer 3f073f261d9447b99220aa532c2b1072: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch)
I20240502 01:58:43.512964 2875 consensus_queue.cc:1035] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 44, Time since last communication: 0.000s
I20240502 01:58:43.516651 2879 raft_consensus.cc:2932] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [term 1 LEADER]: Committing config change with OpId 1.46: config changed from index -1 to 46, VOTER 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:43.517047 1125 raft_consensus.cc:2932] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index -1 to 46, VOTER 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:43.520009 978 catalog_manager.cc:5484] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683 reported cstate change: config changed from index -1 to 46, VOTER 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "3f073f261d9447b99220aa532c2b1072" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:43.528054 2809 heartbeater.cc:503] Master 127.27.111.254:38251 was elected leader, sending a full tablet report...
W20240502 01:58:43.528684 1323 consensus_peers.cc:485] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 -> Peer 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199:33361): Couldn't send request to peer 37279883fafe432f81fe7b8224dbe5fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6da3730570e48d2aba7457e9d41dfcc. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.589735 1125 consensus_queue.cc:237] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.36, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:43.591293 1386 raft_consensus.cc:1261] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20240502 01:58:43.592325 2853 consensus_queue.cc:1035] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 35, Time since last communication: 0.000s
W20240502 01:58:43.598821 1058 consensus_peers.cc:485] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 -> Peer 72de8d04b745434f923594c920aff90c (127.27.111.198:45287): Couldn't send request to peer 72de8d04b745434f923594c920aff90c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90948f77e44241e1883249fbcdcf1515. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.598927 1387 raft_consensus.cc:1203] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.35->[1.36-1.37] Dedup: 1.36->[1.37-1.37]
I20240502 01:58:43.601605 2888 raft_consensus.cc:2932] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Committing config change with OpId 1.37: config changed from index -1 to 37, VOTER 72de8d04b745434f923594c920aff90c (127.27.111.198) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:43.602883 1387 raft_consensus.cc:2932] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, VOTER 72de8d04b745434f923594c920aff90c (127.27.111.198) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:43.607237 978 catalog_manager.cc:5484] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072 reported cstate change: config changed from index -1 to 37, VOTER 72de8d04b745434f923594c920aff90c (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:43.670414 1255 consensus_queue.cc:237] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:43.672087 1125 raft_consensus.cc:1261] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20240502 01:58:43.672971 2064 consensus_queue.cc:1035] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 01:58:43.674623 2064 raft_consensus.cc:2932] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index -1 to 35, VOTER 07c427880bd54215847186eab221fce5 (127.27.111.197) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:43.675311 1125 raft_consensus.cc:2932] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index -1 to 35, VOTER 07c427880bd54215847186eab221fce5 (127.27.111.197) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:43.677706 978 catalog_manager.cc:5484] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683 reported cstate change: config changed from index -1 to 35, VOTER 07c427880bd54215847186eab221fce5 (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
W20240502 01:58:43.681507 1192 consensus_peers.cc:485] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 -> Peer 07c427880bd54215847186eab221fce5 (127.27.111.197:39309): Couldn't send request to peer 07c427880bd54215847186eab221fce5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd52517cbc5b43b798af83233b3759b9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.725706 1386 consensus_queue.cc:237] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.50, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:43.727277 1256 raft_consensus.cc:1261] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Refusing update from remote peer 3f073f261d9447b99220aa532c2b1072: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
W20240502 01:58:43.727671 1323 consensus_peers.cc:485] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 -> Peer 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199:33361): Couldn't send request to peer 37279883fafe432f81fe7b8224dbe5fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8a946d0b927414f88c615d54ce5fc7f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.727977 2014 consensus_queue.cc:1035] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 49, Time since last communication: 0.000s
I20240502 01:58:43.731851 2014 raft_consensus.cc:2932] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [term 1 LEADER]: Committing config change with OpId 1.51: config changed from index -1 to 51, VOTER 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:43.732842 1256 raft_consensus.cc:2932] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, VOTER 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:43.734777 982 catalog_manager.cc:5484] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 reported cstate change: config changed from index -1 to 51, VOTER 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "3f073f261d9447b99220aa532c2b1072" committed_config { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:43.789682 3080 raft_consensus.cc:482] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:58:43.789899 3080 raft_consensus.cc:504] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:43.792639 3080 leader_election.cc:288] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:43.798940 1124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa2250275ba4eb391f88f9ee0fd52e9" candidate_uuid: "07c427880bd54215847186eab221fce5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a571e02bbe774f45b80bb4383642c683" is_pre_election: true
I20240502 01:58:43.799109 1124 raft_consensus.cc:2454] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c427880bd54215847186eab221fce5 in term 1.
I20240502 01:58:43.799714 2432 leader_election.cc:302] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 07c427880bd54215847186eab221fce5, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:43.802778 3080 raft_consensus.cc:2781] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:58:43.802863 3080 raft_consensus.cc:482] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:58:43.802919 3080 raft_consensus.cc:3037] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:43.806353 3080 raft_consensus.cc:504] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:43.807825 3080 leader_election.cc:288] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [CANDIDATE]: Term 2 election: Requested vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:43.807968 1124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fa2250275ba4eb391f88f9ee0fd52e9" candidate_uuid: "07c427880bd54215847186eab221fce5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a571e02bbe774f45b80bb4383642c683"
I20240502 01:58:43.808094 1124 raft_consensus.cc:3037] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:43.809178 1124 raft_consensus.cc:2454] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c427880bd54215847186eab221fce5 in term 2.
I20240502 01:58:43.810046 2432 leader_election.cc:302] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 07c427880bd54215847186eab221fce5, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:43.812537 3080 raft_consensus.cc:2781] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:43.812748 3080 raft_consensus.cc:686] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [term 2 LEADER]: Becoming Leader. State: Replica: 07c427880bd54215847186eab221fce5, State: Running, Role: LEADER
I20240502 01:58:43.813146 3080 consensus_queue.cc:237] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:43.816419 978 catalog_manager.cc:5484] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "07c427880bd54215847186eab221fce5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:43.823664 1255 consensus_queue.cc:237] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.48, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:43.830936 1384 raft_consensus.cc:1261] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240502 01:58:43.832346 2887 consensus_queue.cc:1035] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 47, Time since last communication: 0.000s
I20240502 01:58:43.842806 2887 raft_consensus.cc:2932] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Committing config change with OpId 1.49: config changed from index -1 to 49, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:43.843061 1384 raft_consensus.cc:2932] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index -1 to 49, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
W20240502 01:58:43.845459 1190 consensus_peers.cc:485] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 -> Peer 09401a31cfd046a78a67a2d51e290224 (127.27.111.196:43819): Couldn't send request to peer 09401a31cfd046a78a67a2d51e290224. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 299019dc3d0848ad9324a84e319b9222. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.846503 982 catalog_manager.cc:5484] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072 reported cstate change: config changed from index -1 to 49, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:43.908102 1256 consensus_queue.cc:237] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 1.42, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
W20240502 01:58:43.914022 1189 consensus_peers.cc:485] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 -> Peer 72de8d04b745434f923594c920aff90c (127.27.111.198:45287): Couldn't send request to peer 72de8d04b745434f923594c920aff90c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0e16afa25404fa78e03547f7b140283. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:43.914085 1124 raft_consensus.cc:1261] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 42. Preceding OpId from leader: term: 1 index: 43. (index mismatch)
I20240502 01:58:43.914932 2064 consensus_queue.cc:1035] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 41, Time since last communication: 0.000s
I20240502 01:58:43.920711 2064 raft_consensus.cc:2932] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Committing config change with OpId 1.43: config changed from index -1 to 43, VOTER 72de8d04b745434f923594c920aff90c (127.27.111.198) added. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:43.922425 1123 raft_consensus.cc:2932] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Committing config change with OpId 1.43: config changed from index -1 to 43, VOTER 72de8d04b745434f923594c920aff90c (127.27.111.198) added. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:43.925012 978 catalog_manager.cc:5484] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: config changed from index -1 to 43, VOTER 72de8d04b745434f923594c920aff90c (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:43.961864 3115 ts_tablet_manager.cc:902] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: Initiating tablet copy from peer a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:43.963635 3115 tablet_copy_client.cc:308] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:39473
I20240502 01:58:43.975448 1145 tablet_copy_service.cc:140] P a571e02bbe774f45b80bb4383642c683: Received BeginTabletCopySession request for tablet 690f9ec1b4c34a919a9a09ae7c618ef3 from peer 09401a31cfd046a78a67a2d51e290224 ({username='slave'} at 127.27.111.196:45417)
I20240502 01:58:43.975623 1145 tablet_copy_service.cc:161] P a571e02bbe774f45b80bb4383642c683: Beginning new tablet copy session on tablet 690f9ec1b4c34a919a9a09ae7c618ef3 from peer 09401a31cfd046a78a67a2d51e290224 at {username='slave'} at 127.27.111.196:45417: session id = 09401a31cfd046a78a67a2d51e290224-690f9ec1b4c34a919a9a09ae7c618ef3
I20240502 01:58:43.977085 1145 tablet_copy_source_session.cc:205] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:43.981846 3115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690f9ec1b4c34a919a9a09ae7c618ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.018183 3115 tablet_copy_client.cc:783] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.018494 3115 tablet_copy_client.cc:647] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.024761 3115 tablet_copy_client.cc:523] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.027858 1256 consensus_queue.cc:237] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 1.52, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:44.028898 1387 raft_consensus.cc:1261] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 52. Preceding OpId from leader: term: 1 index: 53. (index mismatch)
I20240502 01:58:44.029346 2938 consensus_queue.cc:1035] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20240502 01:58:44.029853 3115 tablet_bootstrap.cc:492] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: Bootstrap starting.
I20240502 01:58:44.031404 2938 raft_consensus.cc:2932] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Committing config change with OpId 1.53: config changed from index -1 to 53, VOTER 07c427880bd54215847186eab221fce5 (127.27.111.197) added. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:44.033540 1387 raft_consensus.cc:2932] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Committing config change with OpId 1.53: config changed from index -1 to 53, VOTER 07c427880bd54215847186eab221fce5 (127.27.111.197) added. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:44.034679 978 catalog_manager.cc:5484] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: config changed from index -1 to 53, VOTER 07c427880bd54215847186eab221fce5 (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
W20240502 01:58:44.035917 1192 consensus_peers.cc:485] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 -> Peer 07c427880bd54215847186eab221fce5 (127.27.111.197:39309): Couldn't send request to peer 07c427880bd54215847186eab221fce5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a59a5084412f4082bdd35e7c546e71b4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.052572 3115 log.cc:824] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:44.056468 3132 ts_tablet_manager.cc:902] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb: Initiating tablet copy from peer 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:44.058341 3132 tablet_copy_client.cc:308] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:45227
I20240502 01:58:44.061939 3134 ts_tablet_manager.cc:902] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c: Initiating tablet copy from peer a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:44.062790 3134 tablet_copy_client.cc:308] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:39473
I20240502 01:58:44.094079 1407 tablet_copy_service.cc:140] P 3f073f261d9447b99220aa532c2b1072: Received BeginTabletCopySession request for tablet a6da3730570e48d2aba7457e9d41dfcc from peer 37279883fafe432f81fe7b8224dbe5fb ({username='slave'} at 127.27.111.199:58189)
I20240502 01:58:44.094255 1407 tablet_copy_service.cc:161] P 3f073f261d9447b99220aa532c2b1072: Beginning new tablet copy session on tablet a6da3730570e48d2aba7457e9d41dfcc from peer 37279883fafe432f81fe7b8224dbe5fb at {username='slave'} at 127.27.111.199:58189: session id = 37279883fafe432f81fe7b8224dbe5fb-a6da3730570e48d2aba7457e9d41dfcc
I20240502 01:58:44.095800 1407 tablet_copy_source_session.cc:205] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.126297 1145 tablet_copy_service.cc:140] P a571e02bbe774f45b80bb4383642c683: Received BeginTabletCopySession request for tablet 90948f77e44241e1883249fbcdcf1515 from peer 72de8d04b745434f923594c920aff90c ({username='slave'} at 127.27.111.198:58679)
I20240502 01:58:44.126453 1145 tablet_copy_service.cc:161] P a571e02bbe774f45b80bb4383642c683: Beginning new tablet copy session on tablet 90948f77e44241e1883249fbcdcf1515 from peer 72de8d04b745434f923594c920aff90c at {username='slave'} at 127.27.111.198:58679: session id = 72de8d04b745434f923594c920aff90c-90948f77e44241e1883249fbcdcf1515
I20240502 01:58:44.127372 1145 tablet_copy_source_session.cc:205] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.133041 3134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90948f77e44241e1883249fbcdcf1515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.143141 3115 tablet_bootstrap.cc:492] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: Bootstrap replayed 1/1 log segments. Stats: ops{read=51 overwritten=0 applied=51 ignored=0} inserts{seen=723 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:44.143599 3115 tablet_bootstrap.cc:492] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: Bootstrap complete.
I20240502 01:58:44.143965 3115 ts_tablet_manager.cc:1367] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: Time spent bootstrapping tablet: real 0.114s user 0.068s sys 0.006s
I20240502 01:58:44.146798 3115 raft_consensus.cc:348] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:44.147084 3115 raft_consensus.cc:729] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09401a31cfd046a78a67a2d51e290224, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.147933 3115 consensus_queue.cc:260] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 51, Last appended: 1.51, Last appended by leader: 51, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:44.137204 3132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6da3730570e48d2aba7457e9d41dfcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.148891 3115 ts_tablet_manager.cc:1398] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 01:58:44.153841 3134 tablet_copy_client.cc:783] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.154145 3134 tablet_copy_client.cc:647] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.154789 1145 tablet_copy_service.cc:342] P a571e02bbe774f45b80bb4383642c683: Request end of tablet copy session 09401a31cfd046a78a67a2d51e290224-690f9ec1b4c34a919a9a09ae7c618ef3 received from {username='slave'} at 127.27.111.196:45417
I20240502 01:58:44.154876 1145 tablet_copy_service.cc:433] P a571e02bbe774f45b80bb4383642c683: ending tablet copy session 09401a31cfd046a78a67a2d51e290224-690f9ec1b4c34a919a9a09ae7c618ef3 on tablet 690f9ec1b4c34a919a9a09ae7c618ef3 with peer 09401a31cfd046a78a67a2d51e290224
I20240502 01:58:44.158111 3134 tablet_copy_client.cc:523] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.160372 3134 tablet_bootstrap.cc:492] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c: Bootstrap starting.
I20240502 01:58:44.173206 1386 consensus_queue.cc:237] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.50, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:44.175063 1256 raft_consensus.cc:1261] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Refusing update from remote peer 3f073f261d9447b99220aa532c2b1072: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240502 01:58:44.175148 3134 log.cc:824] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:44.175585 2968 consensus_queue.cc:1035] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20240502 01:58:44.179950 2968 raft_consensus.cc:2932] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [term 1 LEADER]: Committing config change with OpId 1.51: config changed from index -1 to 51, VOTER 72de8d04b745434f923594c920aff90c (127.27.111.198) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
W20240502 01:58:44.182873 1320 consensus_peers.cc:485] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 -> Peer 72de8d04b745434f923594c920aff90c (127.27.111.198:45287): Couldn't send request to peer 72de8d04b745434f923594c920aff90c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82337220b48d45fba654828a31eabbf4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.183200 982 catalog_manager.cc:5484] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 reported cstate change: config changed from index -1 to 51, VOTER 72de8d04b745434f923594c920aff90c (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "3f073f261d9447b99220aa532c2b1072" committed_config { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:44.187558 1256 raft_consensus.cc:2932] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, VOTER 72de8d04b745434f923594c920aff90c (127.27.111.198) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:44.196833 3132 tablet_copy_client.cc:783] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.197307 3132 tablet_copy_client.cc:647] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.199544 3132 tablet_copy_client.cc:523] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.205595 3132 tablet_bootstrap.cc:492] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb: Bootstrap starting.
I20240502 01:58:44.218114 3169 ts_tablet_manager.cc:902] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5: Initiating tablet copy from peer 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:44.219267 3169 tablet_copy_client.cc:308] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:33267
I20240502 01:58:44.227017 3132 log.cc:824] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb: Log is configured to *not* fsync() on all Append() calls
I20240502 01:58:44.228261 1276 tablet_copy_service.cc:140] P 4d8c8df1c10b41fab826db1b2ef82e87: Received BeginTabletCopySession request for tablet dd52517cbc5b43b798af83233b3759b9 from peer 07c427880bd54215847186eab221fce5 ({username='slave'} at 127.27.111.197:44231)
I20240502 01:58:44.228425 1276 tablet_copy_service.cc:161] P 4d8c8df1c10b41fab826db1b2ef82e87: Beginning new tablet copy session on tablet dd52517cbc5b43b798af83233b3759b9 from peer 07c427880bd54215847186eab221fce5 at {username='slave'} at 127.27.111.197:44231: session id = 07c427880bd54215847186eab221fce5-dd52517cbc5b43b798af83233b3759b9
I20240502 01:58:44.229907 1276 tablet_copy_source_session.cc:205] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.239024 3169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd52517cbc5b43b798af83233b3759b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.242959 1123 consensus_queue.cc:237] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.51, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:44.243373 3169 tablet_copy_client.cc:783] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.243633 3169 tablet_copy_client.cc:647] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.246466 3169 tablet_copy_client.cc:523] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.246460 1387 raft_consensus.cc:1261] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 1 index: 51. Preceding OpId from leader: term: 1 index: 52. (index mismatch)
I20240502 01:58:44.247736 3038 consensus_queue.cc:1035] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 50, Time since last communication: 0.000s
I20240502 01:58:44.250527 3169 tablet_bootstrap.cc:492] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5: Bootstrap starting.
I20240502 01:58:44.253885 3038 raft_consensus.cc:2932] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Committing config change with OpId 1.52: config changed from index -1 to 52, VOTER 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:44.254557 1386 raft_consensus.cc:2932] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Committing config change with OpId 1.52: config changed from index -1 to 52, VOTER 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:44.261945 982 catalog_manager.cc:5484] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072 reported cstate change: config changed from index -1 to 52, VOTER 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:44.294595 3134 tablet_bootstrap.cc:492] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c: Bootstrap replayed 1/1 log segments. Stats: ops{read=52 overwritten=0 applied=52 ignored=0} inserts{seen=737 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:44.295064 3134 tablet_bootstrap.cc:492] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c: Bootstrap complete.
I20240502 01:58:44.295442 3134 ts_tablet_manager.cc:1367] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c: Time spent bootstrapping tablet: real 0.135s user 0.076s sys 0.000s
I20240502 01:58:44.298243 3134 raft_consensus.cc:348] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:44.305506 3134 raft_consensus.cc:729] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72de8d04b745434f923594c920aff90c, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.306087 3134 consensus_queue.cc:260] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 52, Last appended: 1.52, Last appended by leader: 52, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:44.309774 3134 ts_tablet_manager.cc:1398] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c: Time spent starting tablet: real 0.014s user 0.004s sys 0.001s
I20240502 01:58:44.313207 1144 tablet_copy_service.cc:342] P a571e02bbe774f45b80bb4383642c683: Request end of tablet copy session 72de8d04b745434f923594c920aff90c-90948f77e44241e1883249fbcdcf1515 received from {username='slave'} at 127.27.111.198:58679
I20240502 01:58:44.313302 1144 tablet_copy_service.cc:433] P a571e02bbe774f45b80bb4383642c683: ending tablet copy session 72de8d04b745434f923594c920aff90c-90948f77e44241e1883249fbcdcf1515 on tablet 90948f77e44241e1883249fbcdcf1515 with peer 72de8d04b745434f923594c920aff90c
W20240502 01:58:44.316088 1061 consensus_peers.cc:485] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 -> Peer 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199:33361): Couldn't send request to peer 37279883fafe432f81fe7b8224dbe5fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0cde29604ac4292be1ead0a69aa6908. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.352919 3115 ts_tablet_manager.cc:902] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224: Initiating tablet copy from peer 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:44.353384 3115 tablet_copy_client.cc:308] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:33267
I20240502 01:58:44.355223 3194 ts_tablet_manager.cc:902] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb: Initiating tablet copy from peer 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:44.358203 1276 tablet_copy_service.cc:140] P 4d8c8df1c10b41fab826db1b2ef82e87: Received BeginTabletCopySession request for tablet 299019dc3d0848ad9324a84e319b9222 from peer 09401a31cfd046a78a67a2d51e290224 ({username='slave'} at 127.27.111.196:39791)
I20240502 01:58:44.358307 1276 tablet_copy_service.cc:161] P 4d8c8df1c10b41fab826db1b2ef82e87: Beginning new tablet copy session on tablet 299019dc3d0848ad9324a84e319b9222 from peer 09401a31cfd046a78a67a2d51e290224 at {username='slave'} at 127.27.111.196:39791: session id = 09401a31cfd046a78a67a2d51e290224-299019dc3d0848ad9324a84e319b9222
I20240502 01:58:44.359364 1276 tablet_copy_source_session.cc:205] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.360872 3115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 299019dc3d0848ad9324a84e319b9222. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.363180 3115 tablet_copy_client.cc:783] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.363334 3115 tablet_copy_client.cc:647] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.363685 2367 raft_consensus.cc:1203] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.27->[1.28-1.65] Dedup: 1.51->[1.52-1.65]
I20240502 01:58:44.368912 3115 tablet_copy_client.cc:523] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.370280 3115 tablet_bootstrap.cc:492] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224: Bootstrap starting.
I20240502 01:58:44.391533 1123 raft_consensus.cc:1261] T 0fa2250275ba4eb391f88f9ee0fd52e9 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Refusing update from remote peer 07c427880bd54215847186eab221fce5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:58:44.392359 3205 consensus_queue.cc:1035] T 0fa2250275ba4eb391f88f9ee0fd52e9 P 07c427880bd54215847186eab221fce5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:58:44.402753 3169 tablet_bootstrap.cc:492] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5: Bootstrap replayed 1/1 log segments. Stats: ops{read=59 overwritten=0 applied=59 ignored=0} inserts{seen=731 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:44.403173 3169 tablet_bootstrap.cc:492] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5: Bootstrap complete.
I20240502 01:58:44.403442 3169 ts_tablet_manager.cc:1367] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5: Time spent bootstrapping tablet: real 0.153s user 0.064s sys 0.009s
I20240502 01:58:44.404095 3169 raft_consensus.cc:348] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:44.404230 3169 raft_consensus.cc:729] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c427880bd54215847186eab221fce5, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.404397 3169 consensus_queue.cc:260] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 59, Last appended: 1.59, Last appended by leader: 59, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:44.406447 3169 ts_tablet_manager.cc:1398] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:44.409519 1276 tablet_copy_service.cc:342] P 4d8c8df1c10b41fab826db1b2ef82e87: Request end of tablet copy session 07c427880bd54215847186eab221fce5-dd52517cbc5b43b798af83233b3759b9 received from {username='slave'} at 127.27.111.197:44231
I20240502 01:58:44.409616 1276 tablet_copy_service.cc:433] P 4d8c8df1c10b41fab826db1b2ef82e87: ending tablet copy session 07c427880bd54215847186eab221fce5-dd52517cbc5b43b798af83233b3759b9 on tablet dd52517cbc5b43b798af83233b3759b9 with peer 07c427880bd54215847186eab221fce5
I20240502 01:58:44.424882 3132 tablet_bootstrap.cc:492] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=69 ignored=0} inserts{seen=700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:58:44.425419 3132 tablet_bootstrap.cc:492] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb: Bootstrap complete.
I20240502 01:58:44.425797 3132 ts_tablet_manager.cc:1367] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb: Time spent bootstrapping tablet: real 0.222s user 0.084s sys 0.002s
I20240502 01:58:44.426014 1256 consensus_queue.cc:237] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:44.427965 1385 raft_consensus.cc:1261] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240502 01:58:44.428622 3132 raft_consensus.cc:348] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:44.430210 3132 raft_consensus.cc:729] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37279883fafe432f81fe7b8224dbe5fb, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.430518 2897 consensus_queue.cc:1035] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 70, Time since last communication: 0.000s
I20240502 01:58:44.430758 3132 consensus_queue.cc:260] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:44.431735 3132 ts_tablet_manager.cc:1398] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
I20240502 01:58:44.433609 1407 tablet_copy_service.cc:342] P 3f073f261d9447b99220aa532c2b1072: Request end of tablet copy session 37279883fafe432f81fe7b8224dbe5fb-a6da3730570e48d2aba7457e9d41dfcc received from {username='slave'} at 127.27.111.199:58189
I20240502 01:58:44.433709 1407 tablet_copy_service.cc:433] P 3f073f261d9447b99220aa532c2b1072: ending tablet copy session 37279883fafe432f81fe7b8224dbe5fb-a6da3730570e48d2aba7457e9d41dfcc on tablet a6da3730570e48d2aba7457e9d41dfcc with peer 37279883fafe432f81fe7b8224dbe5fb
I20240502 01:58:44.434720 2929 raft_consensus.cc:2932] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index -1 to 72, VOTER 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199) added. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:44.435012 1384 raft_consensus.cc:2932] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index -1 to 72, VOTER 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199) added. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:44.437266 3194 tablet_copy_client.cc:308] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:45227
W20240502 01:58:44.439607 1192 consensus_peers.cc:485] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 -> Peer 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199:33361): Couldn't send request to peer 37279883fafe432f81fe7b8224dbe5fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 844edd610a734b0994f59a4b0e29af0b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:44.439522 982 catalog_manager.cc:5484] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072 reported cstate change: config changed from index -1 to 72, VOTER 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:44.455173 1407 tablet_copy_service.cc:140] P 3f073f261d9447b99220aa532c2b1072: Received BeginTabletCopySession request for tablet c8a946d0b927414f88c615d54ce5fc7f from peer 37279883fafe432f81fe7b8224dbe5fb ({username='slave'} at 127.27.111.199:58189)
I20240502 01:58:44.455298 1407 tablet_copy_service.cc:161] P 3f073f261d9447b99220aa532c2b1072: Beginning new tablet copy session on tablet c8a946d0b927414f88c615d54ce5fc7f from peer 37279883fafe432f81fe7b8224dbe5fb at {username='slave'} at 127.27.111.199:58189: session id = 37279883fafe432f81fe7b8224dbe5fb-c8a946d0b927414f88c615d54ce5fc7f
I20240502 01:58:44.456254 1407 tablet_copy_source_session.cc:205] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.457233 3194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a946d0b927414f88c615d54ce5fc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.459893 3194 tablet_copy_client.cc:783] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.460091 3194 tablet_copy_client.cc:647] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.461642 3194 tablet_copy_client.cc:523] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.462960 3194 tablet_bootstrap.cc:492] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb: Bootstrap starting.
I20240502 01:58:44.478410 3115 tablet_bootstrap.cc:492] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224: Bootstrap replayed 1/1 log segments. Stats: ops{read=65 overwritten=0 applied=64 ignored=0} inserts{seen=749 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:58:44.478832 3115 tablet_bootstrap.cc:492] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224: Bootstrap complete.
I20240502 01:58:44.479054 3115 ts_tablet_manager.cc:1367] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224: Time spent bootstrapping tablet: real 0.109s user 0.066s sys 0.008s
I20240502 01:58:44.479696 3115 raft_consensus.cc:348] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:44.480302 3115 raft_consensus.cc:729] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09401a31cfd046a78a67a2d51e290224, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.480495 3115 consensus_queue.cc:260] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 64, Last appended: 1.65, Last appended by leader: 65, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:44.482656 3115 ts_tablet_manager.cc:1398] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:58:44.484870 1276 tablet_copy_service.cc:342] P 4d8c8df1c10b41fab826db1b2ef82e87: Request end of tablet copy session 09401a31cfd046a78a67a2d51e290224-299019dc3d0848ad9324a84e319b9222 received from {username='slave'} at 127.27.111.196:39791
I20240502 01:58:44.484956 1276 tablet_copy_service.cc:433] P 4d8c8df1c10b41fab826db1b2ef82e87: ending tablet copy session 09401a31cfd046a78a67a2d51e290224-299019dc3d0848ad9324a84e319b9222 on tablet 299019dc3d0848ad9324a84e319b9222 with peer 09401a31cfd046a78a67a2d51e290224
I20240502 01:58:44.506529 3134 ts_tablet_manager.cc:902] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c: Initiating tablet copy from peer 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:44.507083 3134 tablet_copy_client.cc:308] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:33267
I20240502 01:58:44.513216 1276 tablet_copy_service.cc:140] P 4d8c8df1c10b41fab826db1b2ef82e87: Received BeginTabletCopySession request for tablet c0e16afa25404fa78e03547f7b140283 from peer 72de8d04b745434f923594c920aff90c ({username='slave'} at 127.27.111.198:55529)
I20240502 01:58:44.513360 1276 tablet_copy_service.cc:161] P 4d8c8df1c10b41fab826db1b2ef82e87: Beginning new tablet copy session on tablet c0e16afa25404fa78e03547f7b140283 from peer 72de8d04b745434f923594c920aff90c at {username='slave'} at 127.27.111.198:55529: session id = 72de8d04b745434f923594c920aff90c-c0e16afa25404fa78e03547f7b140283
I20240502 01:58:44.514456 1276 tablet_copy_source_session.cc:205] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.515291 3134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e16afa25404fa78e03547f7b140283. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.517494 3134 tablet_copy_client.cc:783] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.517634 3134 tablet_copy_client.cc:647] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.517572 2763 raft_consensus.cc:1203] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.45->[1.46-1.81] Dedup: 1.70->[1.71-1.81]
I20240502 01:58:44.536828 3134 tablet_copy_client.cc:523] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.538492 3134 tablet_bootstrap.cc:492] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c: Bootstrap starting.
I20240502 01:58:44.574786 3194 tablet_bootstrap.cc:492] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=68 overwritten=0 applied=67 ignored=0} inserts{seen=698 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:58:44.575145 3194 tablet_bootstrap.cc:492] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb: Bootstrap complete.
I20240502 01:58:44.575381 3194 ts_tablet_manager.cc:1367] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb: Time spent bootstrapping tablet: real 0.112s user 0.067s sys 0.005s
I20240502 01:58:44.576052 3194 raft_consensus.cc:348] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:44.576660 3194 raft_consensus.cc:729] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37279883fafe432f81fe7b8224dbe5fb, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.576843 3194 consensus_queue.cc:260] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 67, Last appended: 1.68, Last appended by leader: 68, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:44.581748 3194 ts_tablet_manager.cc:1398] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:58:44.584002 1407 tablet_copy_service.cc:342] P 3f073f261d9447b99220aa532c2b1072: Request end of tablet copy session 37279883fafe432f81fe7b8224dbe5fb-c8a946d0b927414f88c615d54ce5fc7f received from {username='slave'} at 127.27.111.199:58189
I20240502 01:58:44.584091 1407 tablet_copy_service.cc:433] P 3f073f261d9447b99220aa532c2b1072: ending tablet copy session 37279883fafe432f81fe7b8224dbe5fb-c8a946d0b927414f88c615d54ce5fc7f on tablet c8a946d0b927414f88c615d54ce5fc7f with peer 37279883fafe432f81fe7b8224dbe5fb
I20240502 01:58:44.599500 3169 ts_tablet_manager.cc:902] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5: Initiating tablet copy from peer 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:44.599968 3169 tablet_copy_client.cc:308] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:33267
I20240502 01:58:44.602072 1276 tablet_copy_service.cc:140] P 4d8c8df1c10b41fab826db1b2ef82e87: Received BeginTabletCopySession request for tablet a59a5084412f4082bdd35e7c546e71b4 from peer 07c427880bd54215847186eab221fce5 ({username='slave'} at 127.27.111.197:44231)
I20240502 01:58:44.602176 1276 tablet_copy_service.cc:161] P 4d8c8df1c10b41fab826db1b2ef82e87: Beginning new tablet copy session on tablet a59a5084412f4082bdd35e7c546e71b4 from peer 07c427880bd54215847186eab221fce5 at {username='slave'} at 127.27.111.197:44231: session id = 07c427880bd54215847186eab221fce5-a59a5084412f4082bdd35e7c546e71b4
I20240502 01:58:44.603174 1276 tablet_copy_source_session.cc:205] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.605679 3169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59a5084412f4082bdd35e7c546e71b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.607969 3169 tablet_copy_client.cc:783] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.608114 3169 tablet_copy_client.cc:647] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.619230 2631 raft_consensus.cc:1203] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.59] Dedup: 1.52->[1.53-1.59]
I20240502 01:58:44.671217 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:44.677156 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:44.687009 3283 ts_tablet_manager.cc:902] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c: Initiating tablet copy from peer 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227)
I20240502 01:58:44.687559 3283 tablet_copy_client.cc:308] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:45227
I20240502 01:58:44.692831 3134 tablet_bootstrap.cc:492] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c: Bootstrap replayed 1/1 log segments. Stats: ops{read=59 overwritten=0 applied=59 ignored=0} inserts{seen=731 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:44.693152 3134 tablet_bootstrap.cc:492] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c: Bootstrap complete.
I20240502 01:58:44.693341 3134 ts_tablet_manager.cc:1367] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c: Time spent bootstrapping tablet: real 0.155s user 0.055s sys 0.012s
I20240502 01:58:44.693965 3134 raft_consensus.cc:348] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:44.694084 3134 raft_consensus.cc:729] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72de8d04b745434f923594c920aff90c, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.694244 3134 consensus_queue.cc:260] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 59, Last appended: 1.59, Last appended by leader: 59, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:44.695029 1407 tablet_copy_service.cc:140] P 3f073f261d9447b99220aa532c2b1072: Received BeginTabletCopySession request for tablet 82337220b48d45fba654828a31eabbf4 from peer 72de8d04b745434f923594c920aff90c ({username='slave'} at 127.27.111.198:54437)
I20240502 01:58:44.695127 1407 tablet_copy_service.cc:161] P 3f073f261d9447b99220aa532c2b1072: Beginning new tablet copy session on tablet 82337220b48d45fba654828a31eabbf4 from peer 72de8d04b745434f923594c920aff90c at {username='slave'} at 127.27.111.198:54437: session id = 72de8d04b745434f923594c920aff90c-82337220b48d45fba654828a31eabbf4
I20240502 01:58:44.696036 1407 tablet_copy_source_session.cc:205] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.696578 3134 ts_tablet_manager.cc:1398] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:58:44.698575 1276 tablet_copy_service.cc:342] P 4d8c8df1c10b41fab826db1b2ef82e87: Request end of tablet copy session 72de8d04b745434f923594c920aff90c-c0e16afa25404fa78e03547f7b140283 received from {username='slave'} at 127.27.111.198:55529
I20240502 01:58:44.698669 1276 tablet_copy_service.cc:433] P 4d8c8df1c10b41fab826db1b2ef82e87: ending tablet copy session 72de8d04b745434f923594c920aff90c-c0e16afa25404fa78e03547f7b140283 on tablet c0e16afa25404fa78e03547f7b140283 with peer 72de8d04b745434f923594c920aff90c
I20240502 01:58:44.699026 3283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82337220b48d45fba654828a31eabbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.706588 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:44.724045 2763 raft_consensus.cc:1203] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.50->[1.51-1.73] Dedup: 1.68->[1.69-1.73]
I20240502 01:58:44.729420 3283 tablet_copy_client.cc:783] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.729738 3283 tablet_copy_client.cc:647] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.733945 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:44.734673 3169 tablet_copy_client.cc:523] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.736276 3169 tablet_bootstrap.cc:492] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5: Bootstrap starting.
I20240502 01:58:44.737375 3283 tablet_copy_client.cc:523] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.741010 3283 tablet_bootstrap.cc:492] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c: Bootstrap starting.
I20240502 01:58:44.743348 3194 ts_tablet_manager.cc:902] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb: Initiating tablet copy from peer a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:44.770220 3194 tablet_copy_client.cc:308] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:39473
I20240502 01:58:44.774438 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:44.802469 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:44.804049 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:44.830020 3283 tablet_bootstrap.cc:492] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c: Bootstrap replayed 1/1 log segments. Stats: ops{read=66 overwritten=0 applied=66 ignored=0} inserts{seen=697 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:44.830363 3283 tablet_bootstrap.cc:492] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c: Bootstrap complete.
I20240502 01:58:44.830598 3283 ts_tablet_manager.cc:1367] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c: Time spent bootstrapping tablet: real 0.090s user 0.062s sys 0.008s
I20240502 01:58:44.831265 3283 raft_consensus.cc:348] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:44.831388 3283 raft_consensus.cc:729] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72de8d04b745434f923594c920aff90c, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.831543 3283 consensus_queue.cc:260] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 66, Last appended: 1.66, Last appended by leader: 66, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:44.833566 3283 ts_tablet_manager.cc:1398] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:58:44.835319 1407 tablet_copy_service.cc:342] P 3f073f261d9447b99220aa532c2b1072: Request end of tablet copy session 72de8d04b745434f923594c920aff90c-82337220b48d45fba654828a31eabbf4 received from {username='slave'} at 127.27.111.198:54437
I20240502 01:58:44.835408 1407 tablet_copy_service.cc:433] P 3f073f261d9447b99220aa532c2b1072: ending tablet copy session 72de8d04b745434f923594c920aff90c-82337220b48d45fba654828a31eabbf4 on tablet 82337220b48d45fba654828a31eabbf4 with peer 72de8d04b745434f923594c920aff90c
I20240502 01:58:44.836272 2499 raft_consensus.cc:1203] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.34->[1.35-1.73] Dedup: 1.59->[1.60-1.73]
I20240502 01:58:44.883654 1144 tablet_copy_service.cc:140] P a571e02bbe774f45b80bb4383642c683: Received BeginTabletCopySession request for tablet d0cde29604ac4292be1ead0a69aa6908 from peer 37279883fafe432f81fe7b8224dbe5fb ({username='slave'} at 127.27.111.199:58657)
I20240502 01:58:44.883769 1144 tablet_copy_service.cc:161] P a571e02bbe774f45b80bb4383642c683: Beginning new tablet copy session on tablet d0cde29604ac4292be1ead0a69aa6908 from peer 37279883fafe432f81fe7b8224dbe5fb at {username='slave'} at 127.27.111.199:58657: session id = 37279883fafe432f81fe7b8224dbe5fb-d0cde29604ac4292be1ead0a69aa6908
I20240502 01:58:44.884755 1144 tablet_copy_source_session.cc:205] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:44.886202 3194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cde29604ac4292be1ead0a69aa6908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:44.890103 3194 tablet_copy_client.cc:783] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:44.890280 3194 tablet_copy_client.cc:647] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:44.895951 3194 tablet_copy_client.cc:523] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:44.897627 3194 tablet_bootstrap.cc:492] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb: Bootstrap starting.
I20240502 01:58:44.900317 2631 raft_consensus.cc:1203] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.42->[1.43-1.67] Dedup: 1.59->[1.60-1.67]
I20240502 01:58:44.910894 2367 raft_consensus.cc:1203] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.74] Dedup: 1.65->[1.66-1.74]
I20240502 01:58:44.985767 3169 tablet_bootstrap.cc:492] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5: Bootstrap replayed 1/1 log segments. Stats: ops{read=66 overwritten=0 applied=66 ignored=0} inserts{seen=751 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:44.986124 3169 tablet_bootstrap.cc:492] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5: Bootstrap complete.
I20240502 01:58:44.986318 3169 ts_tablet_manager.cc:1367] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5: Time spent bootstrapping tablet: real 0.250s user 0.073s sys 0.001s
I20240502 01:58:44.986800 3169 raft_consensus.cc:348] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:44.987126 3169 raft_consensus.cc:729] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c427880bd54215847186eab221fce5, State: Initialized, Role: FOLLOWER
I20240502 01:58:44.987335 3169 consensus_queue.cc:260] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 66, Last appended: 1.66, Last appended by leader: 66, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:44.987643 3169 ts_tablet_manager.cc:1398] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:58:44.988166 1276 tablet_copy_service.cc:342] P 4d8c8df1c10b41fab826db1b2ef82e87: Request end of tablet copy session 07c427880bd54215847186eab221fce5-a59a5084412f4082bdd35e7c546e71b4 received from {username='slave'} at 127.27.111.197:44231
I20240502 01:58:44.988246 1276 tablet_copy_service.cc:433] P 4d8c8df1c10b41fab826db1b2ef82e87: ending tablet copy session 07c427880bd54215847186eab221fce5-a59a5084412f4082bdd35e7c546e71b4 on tablet a59a5084412f4082bdd35e7c546e71b4 with peer 07c427880bd54215847186eab221fce5
I20240502 01:58:45.004729 3319 ts_tablet_manager.cc:902] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb: Initiating tablet copy from peer 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:45.005517 3319 tablet_copy_client.cc:308] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:33267
I20240502 01:58:45.009320 1276 tablet_copy_service.cc:140] P 4d8c8df1c10b41fab826db1b2ef82e87: Received BeginTabletCopySession request for tablet 844edd610a734b0994f59a4b0e29af0b from peer 37279883fafe432f81fe7b8224dbe5fb ({username='slave'} at 127.27.111.199:43447)
I20240502 01:58:45.009439 1276 tablet_copy_service.cc:161] P 4d8c8df1c10b41fab826db1b2ef82e87: Beginning new tablet copy session on tablet 844edd610a734b0994f59a4b0e29af0b from peer 37279883fafe432f81fe7b8224dbe5fb at {username='slave'} at 127.27.111.199:43447: session id = 37279883fafe432f81fe7b8224dbe5fb-844edd610a734b0994f59a4b0e29af0b
I20240502 01:58:45.010344 1276 tablet_copy_source_session.cc:205] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:45.013224 3319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844edd610a734b0994f59a4b0e29af0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:45.018695 3319 tablet_copy_client.cc:783] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:45.018859 3319 tablet_copy_client.cc:647] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:45.024348 3319 tablet_copy_client.cc:523] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:45.025661 3319 tablet_bootstrap.cc:492] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb: Bootstrap starting.
I20240502 01:58:45.168926 2499 raft_consensus.cc:1203] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.52->[1.53-1.83] Dedup: 1.66->[1.67-1.83]
I20240502 01:58:45.264756 3194 tablet_bootstrap.cc:492] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=72 overwritten=0 applied=72 ignored=0} inserts{seen=703 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:45.265142 3194 tablet_bootstrap.cc:492] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb: Bootstrap complete.
I20240502 01:58:45.265367 3194 ts_tablet_manager.cc:1367] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb: Time spent bootstrapping tablet: real 0.368s user 0.076s sys 0.000s
I20240502 01:58:45.265873 3194 raft_consensus.cc:348] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:45.265985 3194 raft_consensus.cc:729] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37279883fafe432f81fe7b8224dbe5fb, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.287535 3194 consensus_queue.cc:260] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 72, Last appended: 1.72, Last appended by leader: 72, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:45.287988 3194 ts_tablet_manager.cc:1398] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb: Time spent starting tablet: real 0.023s user 0.004s sys 0.000s
I20240502 01:58:45.290059 1144 tablet_copy_service.cc:342] P a571e02bbe774f45b80bb4383642c683: Request end of tablet copy session 37279883fafe432f81fe7b8224dbe5fb-d0cde29604ac4292be1ead0a69aa6908 received from {username='slave'} at 127.27.111.199:58657
I20240502 01:58:45.290143 1144 tablet_copy_service.cc:433] P a571e02bbe774f45b80bb4383642c683: ending tablet copy session 37279883fafe432f81fe7b8224dbe5fb-d0cde29604ac4292be1ead0a69aa6908 on tablet d0cde29604ac4292be1ead0a69aa6908 with peer 37279883fafe432f81fe7b8224dbe5fb
I20240502 01:58:45.290846 2625 raft_consensus.cc:1203] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.50->[1.51-1.83] Dedup: 1.66->[1.67-1.83]
I20240502 01:58:45.315582 3319 tablet_bootstrap.cc:492] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=84 overwritten=0 applied=83 ignored=0} inserts{seen=768 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:58:45.315925 3319 tablet_bootstrap.cc:492] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb: Bootstrap complete.
I20240502 01:58:45.316151 3319 ts_tablet_manager.cc:1367] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb: Time spent bootstrapping tablet: real 0.291s user 0.078s sys 0.009s
I20240502 01:58:45.316895 3319 raft_consensus.cc:348] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:45.317471 3319 raft_consensus.cc:729] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37279883fafe432f81fe7b8224dbe5fb, State: Initialized, Role: FOLLOWER
I20240502 01:58:45.317915 3319 consensus_queue.cc:260] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 83, Last appended: 1.84, Last appended by leader: 84, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:45.318292 3319 ts_tablet_manager.cc:1398] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:58:45.320634 1276 tablet_copy_service.cc:342] P 4d8c8df1c10b41fab826db1b2ef82e87: Request end of tablet copy session 37279883fafe432f81fe7b8224dbe5fb-844edd610a734b0994f59a4b0e29af0b received from {username='slave'} at 127.27.111.199:43447
I20240502 01:58:45.320736 1276 tablet_copy_service.cc:433] P 4d8c8df1c10b41fab826db1b2ef82e87: ending tablet copy session 37279883fafe432f81fe7b8224dbe5fb-844edd610a734b0994f59a4b0e29af0b on tablet 844edd610a734b0994f59a4b0e29af0b with peer 37279883fafe432f81fe7b8224dbe5fb
I20240502 01:58:45.529335 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:45.530756 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:45.560524 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:45.561998 2761 raft_consensus.cc:1203] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.71->[1.72-1.96] Dedup: 1.84->[1.85-1.96]
I20240502 01:58:45.579576 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:45.607049 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:45.619825 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:45.634785 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:45.755046 2762 raft_consensus.cc:1203] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.51->[1.52-1.97] Dedup: 1.72->[1.73-1.97]
I20240502 01:58:45.995193 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:46.027132 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:46.062075 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:46.062575 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:46.084935 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:46.090811 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:46.114907 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:46.246413 1383 consensus_queue.cc:237] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 1.96, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:46.248509 2623 raft_consensus.cc:1261] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Refusing update from remote peer 3f073f261d9447b99220aa532c2b1072: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240502 01:58:46.248981 2968 consensus_queue.cc:1035] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20240502 01:58:46.253258 2014 raft_consensus.cc:2932] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 [term 1 LEADER]: Committing config change with OpId 1.97: config changed from index 51 to 97, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:46.253685 2623 raft_consensus.cc:2932] T 82337220b48d45fba654828a31eabbf4 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 51 to 97, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:46.256110 976 catalog_manager.cc:5484] T 82337220b48d45fba654828a31eabbf4 P 3f073f261d9447b99220aa532c2b1072 reported cstate change: config changed from index 51 to 97, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "3f073f261d9447b99220aa532c2b1072" committed_config { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:46.270784 1234 tablet_service.cc:1514] Processing DeleteTablet for tablet 82337220b48d45fba654828a31eabbf4 with delete_type TABLET_DATA_TOMBSTONED (TS 4d8c8df1c10b41fab826db1b2ef82e87 not found in new config with opid_index 97) from {username='slave'} at 127.0.0.1:40794
I20240502 01:58:46.272306 3442 tablet_replica.cc:330] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87: stopping tablet replica
I20240502 01:58:46.272743 3442 raft_consensus.cc:2229] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:46.273023 3442 raft_consensus.cc:2258] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:46.281786 3442 ts_tablet_manager.cc:1875] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:46.289410 3442 ts_tablet_manager.cc:1888] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.96
I20240502 01:58:46.289644 3442 log.cc:1196] T 82337220b48d45fba654828a31eabbf4 P 4d8c8df1c10b41fab826db1b2ef82e87: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/82337220b48d45fba654828a31eabbf4
I20240502 01:58:46.290454 967 catalog_manager.cc:4830] TS 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267): tablet 82337220b48d45fba654828a31eabbf4 (table rebalance_test_table_0 [id=a79d488e199a471f817d760819b107cd]) successfully deleted
I20240502 01:58:46.542709 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:46.555173 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:46.564571 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:46.583940 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:46.596312 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:46.607005 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:46.626417 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:46.807734 1256 consensus_queue.cc:237] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 1.119, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:46.810276 2498 raft_consensus.cc:1261] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 119. Preceding OpId from leader: term: 1 index: 120. (index mismatch)
I20240502 01:58:46.811064 2837 consensus_queue.cc:1035] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20240502 01:58:46.814170 2837 raft_consensus.cc:2932] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Committing config change with OpId 1.120: config changed from index 53 to 120, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:46.815250 2498 raft_consensus.cc:2932] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Committing config change with OpId 1.120: config changed from index 53 to 120, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:46.817292 982 catalog_manager.cc:5484] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: config changed from index 53 to 120, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:46.906355 1366 tablet_service.cc:1514] Processing DeleteTablet for tablet a59a5084412f4082bdd35e7c546e71b4 with delete_type TABLET_DATA_TOMBSTONED (TS 3f073f261d9447b99220aa532c2b1072 not found in new config with opid_index 120) from {username='slave'} at 127.0.0.1:50588
I20240502 01:58:46.908334 3513 tablet_replica.cc:330] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072: stopping tablet replica
I20240502 01:58:46.909978 3513 raft_consensus.cc:2229] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:46.910172 3513 raft_consensus.cc:2258] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:46.918346 3513 ts_tablet_manager.cc:1875] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:46.982198 3513 ts_tablet_manager.cc:1888] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.119
I20240502 01:58:46.982331 3513 log.cc:1196] T a59a5084412f4082bdd35e7c546e71b4 P 3f073f261d9447b99220aa532c2b1072: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/a59a5084412f4082bdd35e7c546e71b4
I20240502 01:58:46.988909 968 catalog_manager.cc:4830] TS 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227): tablet a59a5084412f4082bdd35e7c546e71b4 (table rebalance_test_table_4 [id=ebb17b8513b14eb0b4a24e8de3c351bf]) successfully deleted
I20240502 01:58:47.214718 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:47.233650 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:47.255532 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:47.281885 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:47.301265 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:47.321543 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:47.343636 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:47.459965 1256 consensus_queue.cc:237] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 1.125, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:47.461097 2631 raft_consensus.cc:1261] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 126. (index mismatch)
I20240502 01:58:47.461689 2938 consensus_queue.cc:1035] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20240502 01:58:47.463855 2937 raft_consensus.cc:2932] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Committing config change with OpId 1.126: config changed from index 43 to 126, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:47.465378 2631 raft_consensus.cc:2932] T c0e16afa25404fa78e03547f7b140283 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index 43 to 126, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:47.468145 981 catalog_manager.cc:5484] T c0e16afa25404fa78e03547f7b140283 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: config changed from index 43 to 126, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:47.517907 1103 tablet_service.cc:1514] Processing DeleteTablet for tablet c0e16afa25404fa78e03547f7b140283 with delete_type TABLET_DATA_TOMBSTONED (TS a571e02bbe774f45b80bb4383642c683 not found in new config with opid_index 126) from {username='slave'} at 127.0.0.1:51778
I20240502 01:58:47.518426 3582 tablet_replica.cc:330] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683: stopping tablet replica
I20240502 01:58:47.518803 3582 raft_consensus.cc:2229] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:47.519026 3582 raft_consensus.cc:2258] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:47.520231 3582 ts_tablet_manager.cc:1875] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:47.523134 3582 ts_tablet_manager.cc:1888] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.125
I20240502 01:58:47.523295 3582 log.cc:1196] T c0e16afa25404fa78e03547f7b140283 P a571e02bbe774f45b80bb4383642c683: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/c0e16afa25404fa78e03547f7b140283
I20240502 01:58:47.525498 968 catalog_manager.cc:4830] TS a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473): tablet c0e16afa25404fa78e03547f7b140283 (table rebalance_test_table_1 [id=e6f3958b51cd48bfb0764c9b91209835]) successfully deleted
I20240502 01:58:47.843909 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:47.864534 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:47.882143 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:47.894835 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:47.922058 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:47.937024 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:47.977375 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:48.240239 1383 consensus_queue.cc:237] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 1.219, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:48.242085 2761 raft_consensus.cc:1261] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Refusing update from remote peer 3f073f261d9447b99220aa532c2b1072: Log matching property violated. Preceding OpId in replica: term: 1 index: 219. Preceding OpId from leader: term: 1 index: 220. (index mismatch)
I20240502 01:58:48.243309 2875 consensus_queue.cc:1035] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20240502 01:58:48.247905 2014 raft_consensus.cc:2932] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 [term 1 LEADER]: Committing config change with OpId 1.220: config changed from index 46 to 220, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:48.249043 2761 raft_consensus.cc:2932] T a6da3730570e48d2aba7457e9d41dfcc P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Committing config change with OpId 1.220: config changed from index 46 to 220, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:48.251031 976 catalog_manager.cc:5484] T a6da3730570e48d2aba7457e9d41dfcc P 3f073f261d9447b99220aa532c2b1072 reported cstate change: config changed from index 46 to 220, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "3f073f261d9447b99220aa532c2b1072" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:48.284245 1103 tablet_service.cc:1514] Processing DeleteTablet for tablet a6da3730570e48d2aba7457e9d41dfcc with delete_type TABLET_DATA_TOMBSTONED (TS a571e02bbe774f45b80bb4383642c683 not found in new config with opid_index 220) from {username='slave'} at 127.0.0.1:51778
I20240502 01:58:48.291616 3661 tablet_replica.cc:330] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683: stopping tablet replica
I20240502 01:58:48.292217 3661 raft_consensus.cc:2229] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:48.292600 3661 raft_consensus.cc:2258] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:48.295951 3661 ts_tablet_manager.cc:1875] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:48.360985 3661 ts_tablet_manager.cc:1888] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.219
I20240502 01:58:48.373458 3661 log.cc:1196] T a6da3730570e48d2aba7457e9d41dfcc P a571e02bbe774f45b80bb4383642c683: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/a6da3730570e48d2aba7457e9d41dfcc
I20240502 01:58:48.374269 968 catalog_manager.cc:4830] TS a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473): tablet a6da3730570e48d2aba7457e9d41dfcc (table rebalance_test_table_3 [id=f35b0e8f48d64f5884fb03539ed2621e]) successfully deleted
I20240502 01:58:48.524703 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:48.547742 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:48.560011 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:48.580262 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:48.586760 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:48.621112 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:48.656163 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:48.785358 1123 consensus_queue.cc:237] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 158, Committed index: 158, Last appended: 1.158, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:48.787155 2367 raft_consensus.cc:1261] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240502 01:58:48.788427 2853 consensus_queue.cc:1035] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 158, Time since last communication: 0.000s
I20240502 01:58:48.790814 2853 raft_consensus.cc:2932] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Committing config change with OpId 1.159: config changed from index 28 to 159, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:48.791383 2366 raft_consensus.cc:2932] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index 28 to 159, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:48.795814 981 catalog_manager.cc:5484] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 reported cstate change: config changed from index 28 to 159, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:48.847391 1234 tablet_service.cc:1514] Processing DeleteTablet for tablet 690f9ec1b4c34a919a9a09ae7c618ef3 with delete_type TABLET_DATA_TOMBSTONED (TS 4d8c8df1c10b41fab826db1b2ef82e87 not found in new config with opid_index 159) from {username='slave'} at 127.0.0.1:40794
I20240502 01:58:48.847831 3730 tablet_replica.cc:330] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87: stopping tablet replica
I20240502 01:58:48.848068 3730 raft_consensus.cc:2229] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:48.848273 3730 raft_consensus.cc:2258] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:48.849510 3730 ts_tablet_manager.cc:1875] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:48.855943 3730 ts_tablet_manager.cc:1888] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.158
I20240502 01:58:48.856113 3730 log.cc:1196] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 4d8c8df1c10b41fab826db1b2ef82e87: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/690f9ec1b4c34a919a9a09ae7c618ef3
I20240502 01:58:48.857123 967 catalog_manager.cc:4830] TS 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267): tablet 690f9ec1b4c34a919a9a09ae7c618ef3 (table rebalance_test_table_4 [id=ebb17b8513b14eb0b4a24e8de3c351bf]) successfully deleted
I20240502 01:58:49.102295 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:49.108470 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:49.120398 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:49.128013 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:49.142601 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:49.195504 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:49.197665 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:49.334801 1123 consensus_queue.cc:237] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 269, Committed index: 269, Last appended: 1.269, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:49.337709 2622 raft_consensus.cc:1261] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 1 index: 269. Preceding OpId from leader: term: 1 index: 270. (index mismatch)
I20240502 01:58:49.338104 2056 consensus_queue.cc:1035] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 270, Last known committed idx: 269, Time since last communication: 0.000s
I20240502 01:58:49.340950 2056 raft_consensus.cc:2932] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Committing config change with OpId 1.270: config changed from index 37 to 270, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:49.342638 2622 raft_consensus.cc:2932] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Committing config change with OpId 1.270: config changed from index 37 to 270, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New config: { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:49.355186 982 catalog_manager.cc:5484] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 reported cstate change: config changed from index 37 to 270, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: 270 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:49.399789 1366 tablet_service.cc:1514] Processing DeleteTablet for tablet 90948f77e44241e1883249fbcdcf1515 with delete_type TABLET_DATA_TOMBSTONED (TS 3f073f261d9447b99220aa532c2b1072 not found in new config with opid_index 270) from {username='slave'} at 127.0.0.1:50588
I20240502 01:58:49.402464 3796 tablet_replica.cc:330] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072: stopping tablet replica
I20240502 01:58:49.403740 3796 raft_consensus.cc:2229] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:49.404348 3796 raft_consensus.cc:2258] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:49.407171 3796 ts_tablet_manager.cc:1875] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:49.410037 3796 ts_tablet_manager.cc:1888] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.269
I20240502 01:58:49.410290 3796 log.cc:1196] T 90948f77e44241e1883249fbcdcf1515 P 3f073f261d9447b99220aa532c2b1072: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/90948f77e44241e1883249fbcdcf1515
I20240502 01:58:49.411417 968 catalog_manager.cc:4830] TS 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227): tablet 90948f77e44241e1883249fbcdcf1515 (table rebalance_test_table_3 [id=f35b0e8f48d64f5884fb03539ed2621e]) successfully deleted
I20240502 01:58:49.575567 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:49.580225 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:49.602996 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:49.649967 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:49.650277 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:49.657049 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:49.706355 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:49.889084 1254 consensus_queue.cc:237] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 1.270, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:49.892230 2498 raft_consensus.cc:1261] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 270. Preceding OpId from leader: term: 1 index: 271. (index mismatch)
I20240502 01:58:49.894109 2938 consensus_queue.cc:1035] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s
I20240502 01:58:49.897804 2938 raft_consensus.cc:2932] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Committing config change with OpId 1.271: config changed from index 35 to 271, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:49.901299 982 catalog_manager.cc:5484] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: config changed from index 35 to 271, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:49.904026 2498 raft_consensus.cc:2932] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Committing config change with OpId 1.271: config changed from index 35 to 271, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:49.950511 1103 tablet_service.cc:1514] Processing DeleteTablet for tablet dd52517cbc5b43b798af83233b3759b9 with delete_type TABLET_DATA_TOMBSTONED (TS a571e02bbe774f45b80bb4383642c683 not found in new config with opid_index 271) from {username='slave'} at 127.0.0.1:51778
I20240502 01:58:49.950865 3854 tablet_replica.cc:330] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683: stopping tablet replica
I20240502 01:58:49.951079 3854 raft_consensus.cc:2229] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:49.951210 3854 raft_consensus.cc:2258] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:49.956907 3854 ts_tablet_manager.cc:1875] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:50.026337 3854 ts_tablet_manager.cc:1888] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.270
I20240502 01:58:50.026567 3854 log.cc:1196] T dd52517cbc5b43b798af83233b3759b9 P a571e02bbe774f45b80bb4383642c683: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/dd52517cbc5b43b798af83233b3759b9
I20240502 01:58:50.028880 968 catalog_manager.cc:4830] TS a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473): tablet dd52517cbc5b43b798af83233b3759b9 (table rebalance_test_table_3 [id=f35b0e8f48d64f5884fb03539ed2621e]) successfully deleted
I20240502 01:58:50.249352 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:50.249614 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:50.269660 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:50.273715 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:50.286576 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:50.299451 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:50.299804 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:50.452677 1384 consensus_queue.cc:237] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 198, Committed index: 198, Last appended: 1.198, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:50.454594 2762 raft_consensus.cc:1261] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Refusing update from remote peer 3f073f261d9447b99220aa532c2b1072: Log matching property violated. Preceding OpId in replica: term: 1 index: 198. Preceding OpId from leader: term: 1 index: 199. (index mismatch)
I20240502 01:58:50.455108 2968 consensus_queue.cc:1035] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 199, Last known committed idx: 198, Time since last communication: 0.000s
I20240502 01:58:50.458770 3859 raft_consensus.cc:2932] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 [term 1 LEADER]: Committing config change with OpId 1.199: config changed from index 51 to 199, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:50.462581 2760 raft_consensus.cc:2932] T c8a946d0b927414f88c615d54ce5fc7f P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Committing config change with OpId 1.199: config changed from index 51 to 199, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:50.464527 980 catalog_manager.cc:5484] T c8a946d0b927414f88c615d54ce5fc7f P 3f073f261d9447b99220aa532c2b1072 reported cstate change: config changed from index 51 to 199, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "3f073f261d9447b99220aa532c2b1072" committed_config { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:50.477505 1234 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a946d0b927414f88c615d54ce5fc7f with delete_type TABLET_DATA_TOMBSTONED (TS 4d8c8df1c10b41fab826db1b2ef82e87 not found in new config with opid_index 199) from {username='slave'} at 127.0.0.1:40794
I20240502 01:58:50.477852 3913 tablet_replica.cc:330] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87: stopping tablet replica
I20240502 01:58:50.478071 3913 raft_consensus.cc:2229] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:50.478250 3913 raft_consensus.cc:2258] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:50.479024 3913 ts_tablet_manager.cc:1875] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:50.482321 3913 ts_tablet_manager.cc:1888] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.198
I20240502 01:58:50.482508 3913 log.cc:1196] T c8a946d0b927414f88c615d54ce5fc7f P 4d8c8df1c10b41fab826db1b2ef82e87: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/c8a946d0b927414f88c615d54ce5fc7f
I20240502 01:58:50.483572 967 catalog_manager.cc:4830] TS 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267): tablet c8a946d0b927414f88c615d54ce5fc7f (table rebalance_test_table_2 [id=2fba41bfdcb948309c7c7ecff39e9c06]) successfully deleted
I20240502 01:58:50.642594 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:50.642930 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:50.643929 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:50.652158 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:50.652413 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:50.653668 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:50.655625 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:50.884502 1254 consensus_queue.cc:237] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 1.223, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:50.885910 2366 raft_consensus.cc:1261] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 223. Preceding OpId from leader: term: 1 index: 224. (index mismatch)
I20240502 01:58:50.887391 2938 consensus_queue.cc:1035] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I20240502 01:58:50.891836 2929 raft_consensus.cc:2932] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Committing config change with OpId 1.224: config changed from index 49 to 224, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:50.892446 2366 raft_consensus.cc:2932] T 299019dc3d0848ad9324a84e319b9222 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Committing config change with OpId 1.224: config changed from index 49 to 224, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:50.894906 982 catalog_manager.cc:5484] T 299019dc3d0848ad9324a84e319b9222 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: config changed from index 49 to 224, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:50.898762 1366 tablet_service.cc:1514] Processing DeleteTablet for tablet 299019dc3d0848ad9324a84e319b9222 with delete_type TABLET_DATA_TOMBSTONED (TS 3f073f261d9447b99220aa532c2b1072 not found in new config with opid_index 224) from {username='slave'} at 127.0.0.1:50588
I20240502 01:58:50.905671 3970 tablet_replica.cc:330] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072: stopping tablet replica
I20240502 01:58:50.905895 3970 raft_consensus.cc:2229] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:50.906085 3970 raft_consensus.cc:2258] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:50.908453 3970 ts_tablet_manager.cc:1875] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:50.926532 3970 ts_tablet_manager.cc:1888] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.223
I20240502 01:58:50.926789 3970 log.cc:1196] T 299019dc3d0848ad9324a84e319b9222 P 3f073f261d9447b99220aa532c2b1072: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/299019dc3d0848ad9324a84e319b9222
I20240502 01:58:50.934463 968 catalog_manager.cc:4830] TS 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227): tablet 299019dc3d0848ad9324a84e319b9222 (table rebalance_test_table_2 [id=2fba41bfdcb948309c7c7ecff39e9c06]) successfully deleted
I20240502 01:58:51.289880 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:51.299160 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:51.301338 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:51.301617 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:51.306744 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:51.327531 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:51.342133 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:51.560362 1254 consensus_queue.cc:237] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 1.253, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:51.561884 2760 raft_consensus.cc:1261] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 253. Preceding OpId from leader: term: 1 index: 254. (index mismatch)
I20240502 01:58:51.562611 3553 consensus_queue.cc:1035] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s
I20240502 01:58:51.564087 2064 raft_consensus.cc:2932] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Committing config change with OpId 1.254: config changed from index 72 to 254, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:51.564429 2760 raft_consensus.cc:2932] T 844edd610a734b0994f59a4b0e29af0b P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Committing config change with OpId 1.254: config changed from index 72 to 254, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:51.566967 978 catalog_manager.cc:5484] T 844edd610a734b0994f59a4b0e29af0b P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: config changed from index 72 to 254, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:51.630376 1366 tablet_service.cc:1514] Processing DeleteTablet for tablet 844edd610a734b0994f59a4b0e29af0b with delete_type TABLET_DATA_TOMBSTONED (TS 3f073f261d9447b99220aa532c2b1072 not found in new config with opid_index 254) from {username='slave'} at 127.0.0.1:50588
I20240502 01:58:51.630785 4051 tablet_replica.cc:330] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072: stopping tablet replica
I20240502 01:58:51.630914 4051 raft_consensus.cc:2229] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:51.631037 4051 raft_consensus.cc:2258] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:51.632695 4051 ts_tablet_manager.cc:1875] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:51.635715 4051 ts_tablet_manager.cc:1888] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.253
I20240502 01:58:51.635874 4051 log.cc:1196] T 844edd610a734b0994f59a4b0e29af0b P 3f073f261d9447b99220aa532c2b1072: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/844edd610a734b0994f59a4b0e29af0b
I20240502 01:58:51.637070 968 catalog_manager.cc:4830] TS 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227): tablet 844edd610a734b0994f59a4b0e29af0b (table rebalance_test_table_0 [id=a79d488e199a471f817d760819b107cd]) successfully deleted
I20240502 01:58:51.812794 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:51.817641 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:51.876315 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:51.877413 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:51.887032 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:51.889266 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:51.890352 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:52.048797 1125 consensus_queue.cc:237] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 1.275, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } }
I20240502 01:58:52.051162 2762 raft_consensus.cc:1261] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 1 index: 275. Preceding OpId from leader: term: 1 index: 276. (index mismatch)
I20240502 01:58:52.051602 2853 consensus_queue.cc:1035] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 276, Last known committed idx: 275, Time since last communication: 0.000s
I20240502 01:58:52.053810 2853 raft_consensus.cc:2932] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Committing config change with OpId 1.276: config changed from index 52 to 276, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:52.060380 982 catalog_manager.cc:5484] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 reported cstate change: config changed from index 52 to 276, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:52.063030 2763 raft_consensus.cc:2932] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Committing config change with OpId 1.276: config changed from index 52 to 276, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } }
I20240502 01:58:52.064604 1366 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cde29604ac4292be1ead0a69aa6908 with delete_type TABLET_DATA_TOMBSTONED (TS 3f073f261d9447b99220aa532c2b1072 not found in new config with opid_index 276) from {username='slave'} at 127.0.0.1:50588
I20240502 01:58:52.064911 4051 tablet_replica.cc:330] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072: stopping tablet replica
I20240502 01:58:52.065058 4051 raft_consensus.cc:2229] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:52.065178 4051 raft_consensus.cc:2258] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:52.066296 4051 ts_tablet_manager.cc:1875] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:52.068661 4051 ts_tablet_manager.cc:1888] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.275
I20240502 01:58:52.068720 4051 log.cc:1196] T d0cde29604ac4292be1ead0a69aa6908 P 3f073f261d9447b99220aa532c2b1072: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/d0cde29604ac4292be1ead0a69aa6908
I20240502 01:58:52.069486 968 catalog_manager.cc:4830] TS 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227): tablet d0cde29604ac4292be1ead0a69aa6908 (table rebalance_test_table_1 [id=e6f3958b51cd48bfb0764c9b91209835]) successfully deleted
I20240502 01:58:52.622442 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:52.632331 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:52.636988 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:52.637908 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:52.638505 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:52.647110 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:52.661599 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:52.888254 1254 consensus_queue.cc:237] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 1.318, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:52.890031 1123 raft_consensus.cc:1261] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 318. Preceding OpId from leader: term: 1 index: 319. (index mismatch)
W20240502 01:58:52.890031 1192 consensus_peers.cc:485] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 -> Peer 07c427880bd54215847186eab221fce5 (127.27.111.197:39309): Couldn't send request to peer 07c427880bd54215847186eab221fce5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa93c869b01b4af69678e378df7db684. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:52.890655 3971 consensus_queue.cc:1035] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20240502 01:58:52.894812 1123 raft_consensus.cc:2932] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Committing config change with OpId 1.319: config changed from index -1 to 319, VOTER 07c427880bd54215847186eab221fce5 (127.27.111.197) added. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:52.896092 3552 raft_consensus.cc:2932] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Committing config change with OpId 1.319: config changed from index -1 to 319, VOTER 07c427880bd54215847186eab221fce5 (127.27.111.197) added. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:52.899037 974 catalog_manager.cc:5484] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: config changed from index -1 to 319, VOTER 07c427880bd54215847186eab221fce5 (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:58:52.953080 1123 consensus_queue.cc:237] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 1.302, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:52.958664 2762 raft_consensus.cc:1261] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
W20240502 01:58:52.959668 1059 consensus_peers.cc:485] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 -> Peer 09401a31cfd046a78a67a2d51e290224 (127.27.111.196:43819): Couldn't send request to peer 09401a31cfd046a78a67a2d51e290224. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0cde29604ac4292be1ead0a69aa6908. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:52.960295 3956 consensus_queue.cc:1035] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240502 01:58:52.963454 3956 raft_consensus.cc:2932] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Committing config change with OpId 1.303: config changed from index 276 to 303, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:52.966320 982 catalog_manager.cc:5484] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 reported cstate change: config changed from index 276 to 303, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:52.967276 2760 raft_consensus.cc:2932] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 276 to 303, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:53.038331 1254 consensus_queue.cc:237] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 1.231, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
W20240502 01:58:53.041055 1189 consensus_peers.cc:485] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 -> Peer 72de8d04b745434f923594c920aff90c (127.27.111.198:45287): Couldn't send request to peer 72de8d04b745434f923594c920aff90c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a59a5084412f4082bdd35e7c546e71b4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:53.042615 2499 raft_consensus.cc:1261] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 1 index: 232. (index mismatch)
I20240502 01:58:53.043849 2064 consensus_queue.cc:1035] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20240502 01:58:53.050056 2064 raft_consensus.cc:2932] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Committing config change with OpId 1.232: config changed from index 120 to 232, VOTER 72de8d04b745434f923594c920aff90c (127.27.111.198) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:53.051730 2499 raft_consensus.cc:2932] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Committing config change with OpId 1.232: config changed from index 120 to 232, VOTER 72de8d04b745434f923594c920aff90c (127.27.111.198) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:53.053095 982 catalog_manager.cc:5484] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: config changed from index 120 to 232, VOTER 72de8d04b745434f923594c920aff90c (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:53.121693 1125 consensus_queue.cc:237] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 408, Committed index: 408, Last appended: 1.409, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:53.123481 2622 raft_consensus.cc:1261] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 1 index: 409. Preceding OpId from leader: term: 1 index: 410. (index mismatch)
I20240502 01:58:53.124266 3978 consensus_queue.cc:1035] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 410, Last known committed idx: 408, Time since last communication: 0.000s
I20240502 01:58:53.127148 3978 raft_consensus.cc:2932] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Committing config change with OpId 1.410: config changed from index 270 to 410, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New config: { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
W20240502 01:58:53.127354 1059 consensus_peers.cc:485] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 -> Peer 09401a31cfd046a78a67a2d51e290224 (127.27.111.196:43819): Couldn't send request to peer 09401a31cfd046a78a67a2d51e290224. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90948f77e44241e1883249fbcdcf1515. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:53.131181 2623 raft_consensus.cc:2932] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Committing config change with OpId 1.410: config changed from index 270 to 410, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New config: { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:53.131853 974 catalog_manager.cc:5484] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 reported cstate change: config changed from index 270 to 410, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:53.219766 1254 consensus_queue.cc:237] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 1.302, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:53.221267 1125 raft_consensus.cc:1261] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
W20240502 01:58:53.221702 1190 consensus_peers.cc:485] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 -> Peer 09401a31cfd046a78a67a2d51e290224 (127.27.111.196:43819): Couldn't send request to peer 09401a31cfd046a78a67a2d51e290224. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 609e71d9a50b4497a0410eb09734c169. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:58:53.221964 3553 consensus_queue.cc:1035] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 301, Time since last communication: 0.000s
I20240502 01:58:53.223475 3972 raft_consensus.cc:2932] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Committing config change with OpId 1.303: config changed from index -1 to 303, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:53.227885 974 catalog_manager.cc:5484] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: config changed from index -1 to 303, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:53.228480 1125 raft_consensus.cc:2932] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index -1 to 303, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) added. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:58:53.443409 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:53.459005 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:53.462563 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:53.468206 4326 ts_tablet_manager.cc:902] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224: Initiating tablet copy from peer a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:53.469023 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:53.469686 4326 tablet_copy_client.cc:308] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:39473
I20240502 01:58:53.470379 1144 tablet_copy_service.cc:140] P a571e02bbe774f45b80bb4383642c683: Received BeginTabletCopySession request for tablet d0cde29604ac4292be1ead0a69aa6908 from peer 09401a31cfd046a78a67a2d51e290224 ({username='slave'} at 127.27.111.196:45417)
I20240502 01:58:53.470544 1144 tablet_copy_service.cc:161] P a571e02bbe774f45b80bb4383642c683: Beginning new tablet copy session on tablet d0cde29604ac4292be1ead0a69aa6908 from peer 09401a31cfd046a78a67a2d51e290224 at {username='slave'} at 127.27.111.196:45417: session id = 09401a31cfd046a78a67a2d51e290224-d0cde29604ac4292be1ead0a69aa6908
I20240502 01:58:53.471562 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:53.471648 1144 tablet_copy_source_session.cc:205] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:53.486155 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:58:53.492136 1192 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.111.197:39309, user_credentials={real_user=slave}} blocked reactor thread for 48046.7us
I20240502 01:58:53.492815 4326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cde29604ac4292be1ead0a69aa6908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.495595 4326 tablet_copy_client.cc:783] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:53.495875 4326 tablet_copy_client.cc:647] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:53.496917 4331 ts_tablet_manager.cc:902] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5: Initiating tablet copy from peer 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:53.497797 4331 tablet_copy_client.cc:308] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:33267
I20240502 01:58:53.498617 4326 tablet_copy_client.cc:523] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:53.498657 1276 tablet_copy_service.cc:140] P 4d8c8df1c10b41fab826db1b2ef82e87: Received BeginTabletCopySession request for tablet fa93c869b01b4af69678e378df7db684 from peer 07c427880bd54215847186eab221fce5 ({username='slave'} at 127.27.111.197:44231)
I20240502 01:58:53.498749 1276 tablet_copy_service.cc:161] P 4d8c8df1c10b41fab826db1b2ef82e87: Beginning new tablet copy session on tablet fa93c869b01b4af69678e378df7db684 from peer 07c427880bd54215847186eab221fce5 at {username='slave'} at 127.27.111.197:44231: session id = 07c427880bd54215847186eab221fce5-fa93c869b01b4af69678e378df7db684
I20240502 01:58:53.499680 1276 tablet_copy_source_session.cc:205] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:53.500020 4326 tablet_bootstrap.cc:492] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224: Bootstrap starting.
I20240502 01:58:53.500525 4331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa93c869b01b4af69678e378df7db684. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.503111 4331 tablet_copy_client.cc:783] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:53.503315 4331 tablet_copy_client.cc:647] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:53.505494 4331 tablet_copy_client.cc:523] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:53.507508 4331 tablet_bootstrap.cc:492] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5: Bootstrap starting.
I20240502 01:58:53.511168 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:53.616318 4339 ts_tablet_manager.cc:902] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c: Initiating tablet copy from peer 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:53.617308 4339 tablet_copy_client.cc:308] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:33267
I20240502 01:58:53.618043 1276 tablet_copy_service.cc:140] P 4d8c8df1c10b41fab826db1b2ef82e87: Received BeginTabletCopySession request for tablet a59a5084412f4082bdd35e7c546e71b4 from peer 72de8d04b745434f923594c920aff90c ({username='slave'} at 127.27.111.198:55529)
I20240502 01:58:53.618148 1276 tablet_copy_service.cc:161] P 4d8c8df1c10b41fab826db1b2ef82e87: Beginning new tablet copy session on tablet a59a5084412f4082bdd35e7c546e71b4 from peer 72de8d04b745434f923594c920aff90c at {username='slave'} at 127.27.111.198:55529: session id = 72de8d04b745434f923594c920aff90c-a59a5084412f4082bdd35e7c546e71b4
I20240502 01:58:53.619129 1276 tablet_copy_source_session.cc:205] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:53.620271 4339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a59a5084412f4082bdd35e7c546e71b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.624372 4339 tablet_copy_client.cc:783] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:53.625181 4339 tablet_copy_client.cc:647] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:53.633229 4339 tablet_copy_client.cc:523] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:53.635820 4339 tablet_bootstrap.cc:492] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c: Bootstrap starting.
I20240502 01:58:53.666568 4344 ts_tablet_manager.cc:902] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224: Initiating tablet copy from peer a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:58:53.671670 4344 tablet_copy_client.cc:308] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:39473
I20240502 01:58:53.672307 1144 tablet_copy_service.cc:140] P a571e02bbe774f45b80bb4383642c683: Received BeginTabletCopySession request for tablet 90948f77e44241e1883249fbcdcf1515 from peer 09401a31cfd046a78a67a2d51e290224 ({username='slave'} at 127.27.111.196:45417)
I20240502 01:58:53.672410 1144 tablet_copy_service.cc:161] P a571e02bbe774f45b80bb4383642c683: Beginning new tablet copy session on tablet 90948f77e44241e1883249fbcdcf1515 from peer 09401a31cfd046a78a67a2d51e290224 at {username='slave'} at 127.27.111.196:45417: session id = 09401a31cfd046a78a67a2d51e290224-90948f77e44241e1883249fbcdcf1515
I20240502 01:58:53.673316 1144 tablet_copy_source_session.cc:205] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:53.676757 4344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90948f77e44241e1883249fbcdcf1515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.700129 4344 tablet_copy_client.cc:783] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:53.700461 4344 tablet_copy_client.cc:647] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:53.713496 4344 tablet_copy_client.cc:523] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:53.727454 4344 tablet_bootstrap.cc:492] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224: Bootstrap starting.
I20240502 01:58:53.791312 4326 tablet_bootstrap.cc:492] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224: Bootstrap replayed 1/1 log segments. Stats: ops{read=316 overwritten=0 applied=316 ignored=0} inserts{seen=945 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:53.791826 4326 tablet_bootstrap.cc:492] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224: Bootstrap complete.
I20240502 01:58:53.792253 4326 ts_tablet_manager.cc:1367] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224: Time spent bootstrapping tablet: real 0.292s user 0.236s sys 0.007s
I20240502 01:58:53.793171 4326 raft_consensus.cc:348] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:53.793411 4326 raft_consensus.cc:729] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09401a31cfd046a78a67a2d51e290224, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.793644 4326 consensus_queue.cc:260] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 316, Last appended: 1.316, Last appended by leader: 316, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:53.794476 4326 ts_tablet_manager.cc:1398] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:53.797250 1144 tablet_copy_service.cc:342] P a571e02bbe774f45b80bb4383642c683: Request end of tablet copy session 09401a31cfd046a78a67a2d51e290224-d0cde29604ac4292be1ead0a69aa6908 received from {username='slave'} at 127.27.111.196:45417
I20240502 01:58:53.797341 1144 tablet_copy_service.cc:433] P a571e02bbe774f45b80bb4383642c683: ending tablet copy session 09401a31cfd046a78a67a2d51e290224-d0cde29604ac4292be1ead0a69aa6908 on tablet d0cde29604ac4292be1ead0a69aa6908 with peer 09401a31cfd046a78a67a2d51e290224
I20240502 01:58:53.829914 4326 ts_tablet_manager.cc:902] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224: Initiating tablet copy from peer 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:53.830471 4326 tablet_copy_client.cc:308] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:33267
I20240502 01:58:53.831408 1276 tablet_copy_service.cc:140] P 4d8c8df1c10b41fab826db1b2ef82e87: Received BeginTabletCopySession request for tablet 609e71d9a50b4497a0410eb09734c169 from peer 09401a31cfd046a78a67a2d51e290224 ({username='slave'} at 127.27.111.196:39791)
I20240502 01:58:53.831516 1276 tablet_copy_service.cc:161] P 4d8c8df1c10b41fab826db1b2ef82e87: Beginning new tablet copy session on tablet 609e71d9a50b4497a0410eb09734c169 from peer 09401a31cfd046a78a67a2d51e290224 at {username='slave'} at 127.27.111.196:39791: session id = 09401a31cfd046a78a67a2d51e290224-609e71d9a50b4497a0410eb09734c169
I20240502 01:58:53.832523 1276 tablet_copy_source_session.cc:205] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:58:53.833681 4326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609e71d9a50b4497a0410eb09734c169. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:58:53.836869 4326 tablet_copy_client.cc:783] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Starting download of 0 data blocks...
I20240502 01:58:53.837316 4326 tablet_copy_client.cc:647] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Starting download of 1 WAL segments...
I20240502 01:58:53.840420 4326 tablet_copy_client.cc:523] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:58:53.842373 4326 tablet_bootstrap.cc:492] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224: Bootstrap starting.
I20240502 01:58:53.919785 4331 tablet_bootstrap.cc:492] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5: Bootstrap replayed 1/1 log segments. Stats: ops{read=342 overwritten=0 applied=341 ignored=0} inserts{seen=1013 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:58:53.920248 4331 tablet_bootstrap.cc:492] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5: Bootstrap complete.
I20240502 01:58:53.920528 4331 ts_tablet_manager.cc:1367] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5: Time spent bootstrapping tablet: real 0.413s user 0.249s sys 0.015s
I20240502 01:58:53.921134 4331 raft_consensus.cc:348] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:53.921521 4331 raft_consensus.cc:729] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c427880bd54215847186eab221fce5, State: Initialized, Role: FOLLOWER
I20240502 01:58:53.921720 4331 consensus_queue.cc:260] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 341, Last appended: 1.342, Last appended by leader: 342, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:53.922094 4331 ts_tablet_manager.cc:1398] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:53.922714 1276 tablet_copy_service.cc:342] P 4d8c8df1c10b41fab826db1b2ef82e87: Request end of tablet copy session 07c427880bd54215847186eab221fce5-fa93c869b01b4af69678e378df7db684 received from {username='slave'} at 127.27.111.197:44231
I20240502 01:58:53.922796 1276 tablet_copy_service.cc:433] P 4d8c8df1c10b41fab826db1b2ef82e87: ending tablet copy session 07c427880bd54215847186eab221fce5-fa93c869b01b4af69678e378df7db684 on tablet fa93c869b01b4af69678e378df7db684 with peer 07c427880bd54215847186eab221fce5
I20240502 01:58:53.944142 2366 raft_consensus.cc:1203] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.302->[1.303-1.335] Dedup: 1.316->[1.317-1.335]
I20240502 01:58:54.033357 2499 raft_consensus.cc:1203] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.318->[1.319-1.353] Dedup: 1.342->[1.343-1.353]
I20240502 01:58:54.183565 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:54.185021 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:54.219535 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:54.232470 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:54.232558 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:54.288437 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:54.302948 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:54.376096 4326 tablet_bootstrap.cc:492] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224: Bootstrap replayed 1/1 log segments. Stats: ops{read=314 overwritten=0 applied=314 ignored=0} inserts{seen=986 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:54.376485 4326 tablet_bootstrap.cc:492] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224: Bootstrap complete.
I20240502 01:58:54.376726 4326 ts_tablet_manager.cc:1367] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224: Time spent bootstrapping tablet: real 0.534s user 0.228s sys 0.019s
I20240502 01:58:54.377228 4326 raft_consensus.cc:348] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:54.377398 4326 raft_consensus.cc:729] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09401a31cfd046a78a67a2d51e290224, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.377569 4326 consensus_queue.cc:260] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 314, Last appended: 1.314, Last appended by leader: 314, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:54.377934 4326 ts_tablet_manager.cc:1398] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:58:54.378852 1276 tablet_copy_service.cc:342] P 4d8c8df1c10b41fab826db1b2ef82e87: Request end of tablet copy session 09401a31cfd046a78a67a2d51e290224-609e71d9a50b4497a0410eb09734c169 received from {username='slave'} at 127.27.111.196:39791
I20240502 01:58:54.378939 1276 tablet_copy_service.cc:433] P 4d8c8df1c10b41fab826db1b2ef82e87: ending tablet copy session 09401a31cfd046a78a67a2d51e290224-609e71d9a50b4497a0410eb09734c169 on tablet 609e71d9a50b4497a0410eb09734c169 with peer 09401a31cfd046a78a67a2d51e290224
I20240502 01:58:54.646198 4344 tablet_bootstrap.cc:492] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224: Bootstrap replayed 1/1 log segments. Stats: ops{read=425 overwritten=0 applied=425 ignored=0} inserts{seen=1108 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:54.646641 4344 tablet_bootstrap.cc:492] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224: Bootstrap complete.
I20240502 01:58:54.647296 4344 ts_tablet_manager.cc:1367] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224: Time spent bootstrapping tablet: real 0.921s user 0.301s sys 0.028s
I20240502 01:58:54.647841 4344 raft_consensus.cc:348] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:54.648350 4344 raft_consensus.cc:729] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09401a31cfd046a78a67a2d51e290224, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.648545 4344 consensus_queue.cc:260] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 425, Last appended: 1.425, Last appended by leader: 425, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:54.651108 4344 ts_tablet_manager.cc:1398] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:58:54.653146 1144 tablet_copy_service.cc:342] P a571e02bbe774f45b80bb4383642c683: Request end of tablet copy session 09401a31cfd046a78a67a2d51e290224-90948f77e44241e1883249fbcdcf1515 received from {username='slave'} at 127.27.111.196:45417
I20240502 01:58:54.653240 1144 tablet_copy_service.cc:433] P a571e02bbe774f45b80bb4383642c683: ending tablet copy session 09401a31cfd046a78a67a2d51e290224-90948f77e44241e1883249fbcdcf1515 on tablet 90948f77e44241e1883249fbcdcf1515 with peer 09401a31cfd046a78a67a2d51e290224
I20240502 01:58:54.748982 2365 raft_consensus.cc:1203] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.302->[1.303-1.333] Dedup: 1.314->[1.315-1.333]
I20240502 01:58:54.750422 4339 tablet_bootstrap.cc:492] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c: Bootstrap replayed 1/1 log segments. Stats: ops{read=252 overwritten=0 applied=252 ignored=0} inserts{seen=935 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:58:54.750824 4339 tablet_bootstrap.cc:492] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c: Bootstrap complete.
I20240502 01:58:54.751063 4339 ts_tablet_manager.cc:1367] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c: Time spent bootstrapping tablet: real 1.115s user 0.211s sys 0.004s
I20240502 01:58:54.751840 4339 raft_consensus.cc:348] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:54.752041 4339 raft_consensus.cc:729] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72de8d04b745434f923594c920aff90c, State: Initialized, Role: FOLLOWER
I20240502 01:58:54.752197 4339 consensus_queue.cc:260] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 252, Last appended: 1.252, Last appended by leader: 252, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:54.752630 4339 ts_tablet_manager.cc:1398] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:58:54.756572 1276 tablet_copy_service.cc:342] P 4d8c8df1c10b41fab826db1b2ef82e87: Request end of tablet copy session 72de8d04b745434f923594c920aff90c-a59a5084412f4082bdd35e7c546e71b4 received from {username='slave'} at 127.27.111.198:55529
I20240502 01:58:54.756664 1276 tablet_copy_service.cc:433] P 4d8c8df1c10b41fab826db1b2ef82e87: ending tablet copy session 72de8d04b745434f923594c920aff90c-a59a5084412f4082bdd35e7c546e71b4 on tablet a59a5084412f4082bdd35e7c546e71b4 with peer 72de8d04b745434f923594c920aff90c
I20240502 01:58:55.037362 2366 raft_consensus.cc:1203] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.409->[1.410-1.469] Dedup: 1.425->[1.426-1.469]
I20240502 01:58:55.096372 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:55.116627 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:55.136440 2625 raft_consensus.cc:1203] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.231->[1.232-1.277] Dedup: 1.252->[1.253-1.277]
I20240502 01:58:55.201315 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:55.215773 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:55.216398 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:58:55.216778 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:55.253016 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:55.371418 1125 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90948f77e44241e1883249fbcdcf1515"
dest_uuid: "a571e02bbe774f45b80bb4383642c683"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52856
I20240502 01:58:55.375542 1125 raft_consensus.cc:595] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Received request to transfer leadership
W20240502 01:58:55.402019 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.405840 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.411998 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.420691 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.429472 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.436569 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.447167 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.458083 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.472726 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.486920 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.502276 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.522936 1085 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.538619 1085 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
I20240502 01:58:55.542572 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:55.544845 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:55.554942 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
I20240502 01:58:55.556649 2608 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:55.557914 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:55.564703 1103 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:58:55.570207 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
I20240502 01:58:55.576190 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:55.596477 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.620200 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.642314 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.663990 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.684160 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
I20240502 01:58:55.687214 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:58:55.706910 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.732905 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.767132 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.793584 1085 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.822582 1085 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
W20240502 01:58:55.850199 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
I20240502 01:58:55.880023 4488 raft_consensus.cc:979] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683: : Instructing follower 72de8d04b745434f923594c920aff90c to start an election
I20240502 01:58:55.880231 4488 raft_consensus.cc:1067] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Signalling peer 72de8d04b745434f923594c920aff90c to start an election
W20240502 01:58:55.880383 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Service unavailable: leader transfer in progress
I20240502 01:58:55.881340 2625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90948f77e44241e1883249fbcdcf1515"
dest_uuid: "72de8d04b745434f923594c920aff90c"
from {username='slave'} at 127.27.111.193:55265
I20240502 01:58:55.881546 2625 raft_consensus.cc:482] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:55.881677 2625 raft_consensus.cc:3037] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:55.883061 2625 raft_consensus.cc:504] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:55.884313 2625 leader_election.cc:288] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [CANDIDATE]: Term 2 election: Requested vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473), 09401a31cfd046a78a67a2d51e290224 (127.27.111.196:43819)
I20240502 01:58:55.887113 1124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90948f77e44241e1883249fbcdcf1515" candidate_uuid: "72de8d04b745434f923594c920aff90c" candidate_term: 2 candidate_status { last_received { term: 1 index: 476 } } ignore_live_leader: true dest_uuid: "a571e02bbe774f45b80bb4383642c683"
I20240502 01:58:55.887243 1124 raft_consensus.cc:3032] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:55.887286 1124 raft_consensus.cc:729] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Running, Role: LEADER
I20240502 01:58:55.887446 1124 consensus_queue.cc:260] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 476, Committed index: 476, Last appended: 1.476, Last appended by leader: 476, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:55.887620 1124 raft_consensus.cc:3037] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:55.888797 1124 raft_consensus.cc:2454] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72de8d04b745434f923594c920aff90c in term 2.
I20240502 01:58:55.893240 2566 leader_election.cc:302] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72de8d04b745434f923594c920aff90c, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:55.896634 4547 raft_consensus.cc:2781] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:55.896968 4547 raft_consensus.cc:686] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [term 2 LEADER]: Becoming Leader. State: Replica: 72de8d04b745434f923594c920aff90c, State: Running, Role: LEADER
I20240502 01:58:55.899029 4547 consensus_queue.cc:237] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 476, Committed index: 476, Last appended: 1.476, Last appended by leader: 476, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:55.906939 974 catalog_manager.cc:5484] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c reported cstate change: term changed from 1 to 2, leader changed from a571e02bbe774f45b80bb4383642c683 (127.27.111.193) to 72de8d04b745434f923594c920aff90c (127.27.111.198). New cstate: current_term: 2 leader_uuid: "72de8d04b745434f923594c920aff90c" committed_config { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
W20240502 01:58:55.912009 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Illegal state: replica a571e02bbe774f45b80bb4383642c683 is not leader of this config: current role FOLLOWER
I20240502 01:58:55.913105 2364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90948f77e44241e1883249fbcdcf1515" candidate_uuid: "72de8d04b745434f923594c920aff90c" candidate_term: 2 candidate_status { last_received { term: 1 index: 476 } } ignore_live_leader: true dest_uuid: "09401a31cfd046a78a67a2d51e290224"
I20240502 01:58:55.913311 2364 raft_consensus.cc:3037] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:55.914834 2364 raft_consensus.cc:2454] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72de8d04b745434f923594c920aff90c in term 2.
W20240502 01:58:55.978097 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39680: Illegal state: replica a571e02bbe774f45b80bb4383642c683 is not leader of this config: current role FOLLOWER
I20240502 01:58:56.023727 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:56.024541 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:56.049054 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:56.050029 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:56.057929 2364 raft_consensus.cc:1261] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224 [term 2 FOLLOWER]: Refusing update from remote peer 72de8d04b745434f923594c920aff90c: Log matching property violated. Preceding OpId in replica: term: 1 index: 476. Preceding OpId from leader: term: 2 index: 478. (index mismatch)
I20240502 01:58:56.058423 1125 raft_consensus.cc:1261] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Refusing update from remote peer 72de8d04b745434f923594c920aff90c: Log matching property violated. Preceding OpId in replica: term: 1 index: 476. Preceding OpId from leader: term: 2 index: 478. (index mismatch)
I20240502 01:58:56.058723 4547 consensus_queue.cc:1035] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 477, Last known committed idx: 476, Time since last communication: 0.000s
I20240502 01:58:56.060645 4547 consensus_queue.cc:1035] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 477, Last known committed idx: 476, Time since last communication: 0.000s
I20240502 01:58:56.061651 3516 mvcc.cc:205] Tried to move back new op lower bound from 7023063597288308736 to 7023063596665622528. Current Snapshot: MvccSnapshot[applied={T|T < 7023063597288308736}]
I20240502 01:58:56.143973 2608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:56.150164 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:56.267403 1103 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:56.392052 1254 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a59a5084412f4082bdd35e7c546e71b4"
dest_uuid: "4d8c8df1c10b41fab826db1b2ef82e87"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41628
I20240502 01:58:56.392246 1254 raft_consensus.cc:595] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Received request to transfer leadership
W20240502 01:58:56.547926 1216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
W20240502 01:58:56.553529 1216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
W20240502 01:58:56.562870 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
W20240502 01:58:56.571462 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
W20240502 01:58:56.577741 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
W20240502 01:58:56.588905 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
W20240502 01:58:56.599846 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
W20240502 01:58:56.611562 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
W20240502 01:58:56.624987 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
W20240502 01:58:56.638911 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
W20240502 01:58:56.651880 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
W20240502 01:58:56.669049 1216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
I20240502 01:58:56.678661 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:56.683321 1216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
I20240502 01:58:56.697964 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:58:56.702059 1213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
I20240502 01:58:56.702848 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:56.708613 1234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:56.711220 2608 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:56.724727 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:56.725836 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
I20240502 01:58:56.733064 1103 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:58:56.746987 1213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
W20240502 01:58:56.766510 1213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
W20240502 01:58:56.792244 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
I20240502 01:58:56.813716 3971 raft_consensus.cc:979] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87: : Instructing follower 72de8d04b745434f923594c920aff90c to start an election
I20240502 01:58:56.813861 3971 raft_consensus.cc:1067] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Signalling peer 72de8d04b745434f923594c920aff90c to start an election
W20240502 01:58:56.815356 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Service unavailable: leader transfer in progress
I20240502 01:58:56.816278 2622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a59a5084412f4082bdd35e7c546e71b4"
dest_uuid: "72de8d04b745434f923594c920aff90c"
from {username='slave'} at 127.27.111.194:33425
I20240502 01:58:56.816417 2622 raft_consensus.cc:482] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:56.816475 2622 raft_consensus.cc:3037] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:56.817659 2622 raft_consensus.cc:504] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:56.818094 2622 leader_election.cc:288] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [CANDIDATE]: Term 2 election: Requested vote from peers 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267), 07c427880bd54215847186eab221fce5 (127.27.111.197:39309)
I20240502 01:58:56.822888 1254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59a5084412f4082bdd35e7c546e71b4" candidate_uuid: "72de8d04b745434f923594c920aff90c" candidate_term: 2 candidate_status { last_received { term: 1 index: 301 } } ignore_live_leader: true dest_uuid: "4d8c8df1c10b41fab826db1b2ef82e87"
I20240502 01:58:56.823022 1254 raft_consensus.cc:3032] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:56.823068 1254 raft_consensus.cc:729] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Running, Role: LEADER
I20240502 01:58:56.823220 1254 consensus_queue.cc:260] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 1.301, Last appended by leader: 301, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:56.823397 1254 raft_consensus.cc:3037] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:56.824596 1254 raft_consensus.cc:2454] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72de8d04b745434f923594c920aff90c in term 2.
I20240502 01:58:56.825505 2565 leader_election.cc:302] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d8c8df1c10b41fab826db1b2ef82e87, 72de8d04b745434f923594c920aff90c; no voters:
I20240502 01:58:56.827507 4547 raft_consensus.cc:2781] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:56.827666 4547 raft_consensus.cc:686] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [term 2 LEADER]: Becoming Leader. State: Replica: 72de8d04b745434f923594c920aff90c, State: Running, Role: LEADER
I20240502 01:58:56.827801 4547 consensus_queue.cc:237] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 1.301, Last appended by leader: 301, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:56.830689 974 catalog_manager.cc:5484] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c reported cstate change: term changed from 1 to 2, leader changed from 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) to 72de8d04b745434f923594c920aff90c (127.27.111.198). New cstate: current_term: 2 leader_uuid: "72de8d04b745434f923594c920aff90c" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
W20240502 01:58:56.841094 1213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33496: Illegal state: replica 4d8c8df1c10b41fab826db1b2ef82e87 is not leader of this config: current role FOLLOWER
I20240502 01:58:56.841102 2496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a59a5084412f4082bdd35e7c546e71b4" candidate_uuid: "72de8d04b745434f923594c920aff90c" candidate_term: 2 candidate_status { last_received { term: 1 index: 301 } } ignore_live_leader: true dest_uuid: "07c427880bd54215847186eab221fce5"
I20240502 01:58:56.841306 2496 raft_consensus.cc:3037] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:56.843518 2496 raft_consensus.cc:2454] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72de8d04b745434f923594c920aff90c in term 2.
I20240502 01:58:56.914378 2497 raft_consensus.cc:1261] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5 [term 2 FOLLOWER]: Refusing update from remote peer 72de8d04b745434f923594c920aff90c: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 2 index: 303. (index mismatch)
I20240502 01:58:56.915148 4547 consensus_queue.cc:1035] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20240502 01:58:56.917496 1254 raft_consensus.cc:1261] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Refusing update from remote peer 72de8d04b745434f923594c920aff90c: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 2 index: 303. (index mismatch)
I20240502 01:58:56.918908 4579 mvcc.cc:205] Tried to move back new op lower bound from 7023063600793571328 to 7023063600448208896. Current Snapshot: MvccSnapshot[applied={T|T < 7023063600793571328}]
I20240502 01:58:56.921778 4634 consensus_queue.cc:1035] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20240502 01:58:56.928835 2826 mvcc.cc:205] Tried to move back new op lower bound from 7023063600793571328 to 7023063600448208896. Current Snapshot: MvccSnapshot[applied={T|T < 7023063600793571328}]
I20240502 01:58:57.032617 2623 consensus_queue.cc:237] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 2.305, Last appended by leader: 301, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } }
I20240502 01:58:57.034842 2497 raft_consensus.cc:1261] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5 [term 2 FOLLOWER]: Refusing update from remote peer 72de8d04b745434f923594c920aff90c: Log matching property violated. Preceding OpId in replica: term: 2 index: 305. Preceding OpId from leader: term: 2 index: 306. (index mismatch)
I20240502 01:58:57.035295 4580 consensus_queue.cc:1035] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20240502 01:58:57.039623 4634 raft_consensus.cc:2932] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c [term 2 LEADER]: Committing config change with OpId 2.306: config changed from index 232 to 306, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:57.041252 2497 raft_consensus.cc:2932] T a59a5084412f4082bdd35e7c546e71b4 P 07c427880bd54215847186eab221fce5 [term 2 FOLLOWER]: Committing config change with OpId 2.306: config changed from index 232 to 306, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:57.043001 974 catalog_manager.cc:5484] T a59a5084412f4082bdd35e7c546e71b4 P 72de8d04b745434f923594c920aff90c reported cstate change: config changed from index 232 to 306, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New cstate: current_term: 2 leader_uuid: "72de8d04b745434f923594c920aff90c" committed_config { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } }
I20240502 01:58:57.047703 1234 tablet_service.cc:1514] Processing DeleteTablet for tablet a59a5084412f4082bdd35e7c546e71b4 with delete_type TABLET_DATA_TOMBSTONED (TS 4d8c8df1c10b41fab826db1b2ef82e87 not found in new config with opid_index 306) from {username='slave'} at 127.0.0.1:40794
I20240502 01:58:57.054107 4665 tablet_replica.cc:330] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87: stopping tablet replica
I20240502 01:58:57.055675 4665 raft_consensus.cc:2229] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:58:57.056586 4665 raft_consensus.cc:2258] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:58:57.059469 4665 ts_tablet_manager.cc:1875] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:58:57.104041 4665 ts_tablet_manager.cc:1888] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.305
I20240502 01:58:57.104274 4665 log.cc:1196] T a59a5084412f4082bdd35e7c546e71b4 P 4d8c8df1c10b41fab826db1b2ef82e87: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/a59a5084412f4082bdd35e7c546e71b4
I20240502 01:58:57.105252 967 catalog_manager.cc:4830] TS 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267): tablet a59a5084412f4082bdd35e7c546e71b4 (table rebalance_test_table_4 [id=ebb17b8513b14eb0b4a24e8de3c351bf]) successfully deleted
I20240502 01:58:57.246533 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:57.252391 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:57.284735 1103 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:57.287930 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:57.289525 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:57.289619 1234 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:58:57.304664 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:57.447396 1124 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0cde29604ac4292be1ead0a69aa6908"
dest_uuid: "a571e02bbe774f45b80bb4383642c683"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52916
I20240502 01:58:57.447569 1124 raft_consensus.cc:595] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Received request to transfer leadership
W20240502 01:58:57.462980 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.470009 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.473543 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.479967 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.490917 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.498123 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.509200 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.523927 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.533985 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.547492 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.564744 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.580080 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.598978 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
I20240502 01:58:57.617061 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:57.619815 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
I20240502 01:58:57.631845 1234 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:58:57.633651 2347 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:57.643483 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
I20240502 01:58:57.645401 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:57.660215 1103 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:57.662173 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:57.662807 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
I20240502 01:58:57.664675 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:58:57.683640 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.706423 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.728493 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.753122 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.778419 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.803853 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.829521 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.855839 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.884640 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
W20240502 01:58:57.912371 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
I20240502 01:58:57.932663 4392 raft_consensus.cc:979] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683: : Instructing follower 09401a31cfd046a78a67a2d51e290224 to start an election
I20240502 01:58:57.932801 4392 raft_consensus.cc:1067] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Signalling peer 09401a31cfd046a78a67a2d51e290224 to start an election
I20240502 01:58:57.936842 2367 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0cde29604ac4292be1ead0a69aa6908"
dest_uuid: "09401a31cfd046a78a67a2d51e290224"
from {username='slave'} at 127.27.111.193:44517
I20240502 01:58:57.937486 2367 raft_consensus.cc:482] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:57.937644 2367 raft_consensus.cc:3037] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:57.939095 2367 raft_consensus.cc:504] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
W20240502 01:58:57.940201 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Service unavailable: leader transfer in progress
I20240502 01:58:57.952863 2367 leader_election.cc:288] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [CANDIDATE]: Term 2 election: Requested vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473), 37279883fafe432f81fe7b8224dbe5fb (127.27.111.199:33361)
I20240502 01:58:57.953541 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cde29604ac4292be1ead0a69aa6908" candidate_uuid: "09401a31cfd046a78a67a2d51e290224" candidate_term: 2 candidate_status { last_received { term: 1 index: 446 } } ignore_live_leader: true dest_uuid: "a571e02bbe774f45b80bb4383642c683"
I20240502 01:58:57.953678 1125 raft_consensus.cc:3032] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:57.953720 1125 raft_consensus.cc:729] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a571e02bbe774f45b80bb4383642c683, State: Running, Role: LEADER
I20240502 01:58:57.954953 1125 consensus_queue.cc:260] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 446, Committed index: 446, Last appended: 1.446, Last appended by leader: 446, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:57.955148 1125 raft_consensus.cc:3037] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:57.956393 1125 raft_consensus.cc:2454] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09401a31cfd046a78a67a2d51e290224 in term 2.
I20240502 01:58:57.956775 2301 leader_election.cc:302] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09401a31cfd046a78a67a2d51e290224, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:57.964082 4770 raft_consensus.cc:2781] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:57.964320 4770 raft_consensus.cc:686] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [term 2 LEADER]: Becoming Leader. State: Replica: 09401a31cfd046a78a67a2d51e290224, State: Running, Role: LEADER
I20240502 01:58:57.964684 4770 consensus_queue.cc:237] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 446, Committed index: 446, Last appended: 1.446, Last appended by leader: 446, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:57.971170 980 catalog_manager.cc:5484] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 reported cstate change: term changed from 1 to 2, leader changed from a571e02bbe774f45b80bb4383642c683 (127.27.111.193) to 09401a31cfd046a78a67a2d51e290224 (127.27.111.196). New cstate: current_term: 2 leader_uuid: "09401a31cfd046a78a67a2d51e290224" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
W20240502 01:58:57.971300 1084 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39664: Illegal state: replica a571e02bbe774f45b80bb4383642c683 is not leader of this config: current role FOLLOWER
I20240502 01:58:57.981761 2762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cde29604ac4292be1ead0a69aa6908" candidate_uuid: "09401a31cfd046a78a67a2d51e290224" candidate_term: 2 candidate_status { last_received { term: 1 index: 446 } } ignore_live_leader: true dest_uuid: "37279883fafe432f81fe7b8224dbe5fb"
I20240502 01:58:57.982012 2762 raft_consensus.cc:3037] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:57.983441 2762 raft_consensus.cc:2454] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09401a31cfd046a78a67a2d51e290224 in term 2.
I20240502 01:58:58.126951 2763 raft_consensus.cc:1261] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb [term 2 FOLLOWER]: Refusing update from remote peer 09401a31cfd046a78a67a2d51e290224: Log matching property violated. Preceding OpId in replica: term: 1 index: 446. Preceding OpId from leader: term: 2 index: 448. (index mismatch)
I20240502 01:58:58.128690 4770 consensus_queue.cc:1035] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I20240502 01:58:58.129184 1123 raft_consensus.cc:1261] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Refusing update from remote peer 09401a31cfd046a78a67a2d51e290224: Log matching property violated. Preceding OpId in replica: term: 1 index: 446. Preceding OpId from leader: term: 2 index: 448. (index mismatch)
I20240502 01:58:58.131080 4770 consensus_queue.cc:1035] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 447, Last known committed idx: 446, Time since last communication: 0.000s
I20240502 01:58:58.131970 4436 mvcc.cc:205] Tried to move back new op lower bound from 7023063605761716224 to 7023063605120045056. Current Snapshot: MvccSnapshot[applied={T|T < 7023063605761716224}]
I20240502 01:58:58.132771 4426 mvcc.cc:205] Tried to move back new op lower bound from 7023063605761716224 to 7023063605120045056. Current Snapshot: MvccSnapshot[applied={T|T < 7023063605761716224 or (T in {7023063605761716224})}]
I20240502 01:58:58.134600 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:58.151754 1234 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:58:58.160730 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:58.174118 2347 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:58.223129 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:58.283151 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:58.330346 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:58.418308 1254 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa93c869b01b4af69678e378df7db684"
dest_uuid: "4d8c8df1c10b41fab826db1b2ef82e87"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41696
I20240502 01:58:58.418511 1254 raft_consensus.cc:595] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Received request to transfer leadership
W20240502 01:58:58.477916 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.484663 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.489761 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.495164 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.503659 1213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.511701 1213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.522504 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.535614 1216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.546416 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.572245 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.588598 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.608593 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.626242 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.657447 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
I20240502 01:58:58.663419 2479 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:58.667717 2347 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:58.678406 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
I20240502 01:58:58.687031 1234 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 01:58:58.699124 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
I20240502 01:58:58.700266 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:58.704797 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:58.710723 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 01:58:58.718554 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
I20240502 01:58:58.722237 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 01:58:58.740222 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.761476 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.781639 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.805321 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.831286 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.858950 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.886122 1215 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
W20240502 01:58:58.912967 1214 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Service unavailable: leader transfer in progress
I20240502 01:58:58.915863 2929 raft_consensus.cc:979] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87: : Instructing follower 07c427880bd54215847186eab221fce5 to start an election
I20240502 01:58:58.915985 2929 raft_consensus.cc:1067] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Signalling peer 07c427880bd54215847186eab221fce5 to start an election
I20240502 01:58:58.917093 2497 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa93c869b01b4af69678e378df7db684"
dest_uuid: "07c427880bd54215847186eab221fce5"
from {username='slave'} at 127.27.111.194:39551
I20240502 01:58:58.917289 2497 raft_consensus.cc:482] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:58.917385 2497 raft_consensus.cc:3037] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:58.918596 2497 raft_consensus.cc:504] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:58.919075 2497 leader_election.cc:288] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [CANDIDATE]: Term 2 election: Requested vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473), 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:58:58.919777 1254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa93c869b01b4af69678e378df7db684" candidate_uuid: "07c427880bd54215847186eab221fce5" candidate_term: 2 candidate_status { last_received { term: 1 index: 471 } } ignore_live_leader: true dest_uuid: "4d8c8df1c10b41fab826db1b2ef82e87"
I20240502 01:58:58.919778 1123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa93c869b01b4af69678e378df7db684" candidate_uuid: "07c427880bd54215847186eab221fce5" candidate_term: 2 candidate_status { last_received { term: 1 index: 471 } } ignore_live_leader: true dest_uuid: "a571e02bbe774f45b80bb4383642c683"
I20240502 01:58:58.919904 1123 raft_consensus.cc:3037] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:58.919903 1254 raft_consensus.cc:3032] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:58.919950 1254 raft_consensus.cc:729] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Running, Role: LEADER
I20240502 01:58:58.920083 1254 consensus_queue.cc:260] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 471, Committed index: 471, Last appended: 1.471, Last appended by leader: 471, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:58.920238 1254 raft_consensus.cc:3037] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:58.921061 1123 raft_consensus.cc:2454] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c427880bd54215847186eab221fce5 in term 2.
I20240502 01:58:58.921234 1254 raft_consensus.cc:2454] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c427880bd54215847186eab221fce5 in term 2.
I20240502 01:58:58.921428 2432 leader_election.cc:302] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c427880bd54215847186eab221fce5, a571e02bbe774f45b80bb4383642c683; no voters:
I20240502 01:58:58.921640 4779 raft_consensus.cc:2781] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:58.921830 4779 raft_consensus.cc:686] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [term 2 LEADER]: Becoming Leader. State: Replica: 07c427880bd54215847186eab221fce5, State: Running, Role: LEADER
I20240502 01:58:58.921986 4779 consensus_queue.cc:237] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 471, Committed index: 471, Last appended: 1.471, Last appended by leader: 471, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:58:58.924079 980 catalog_manager.cc:5484] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 reported cstate change: term changed from 1 to 2, leader changed from 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) to 07c427880bd54215847186eab221fce5 (127.27.111.197). New cstate: current_term: 2 leader_uuid: "07c427880bd54215847186eab221fce5" committed_config { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
W20240502 01:58:58.941759 1216 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33486: Illegal state: replica 4d8c8df1c10b41fab826db1b2ef82e87 is not leader of this config: current role FOLLOWER
W20240502 01:58:59.006772 1085 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52962: Illegal state: replica a571e02bbe774f45b80bb4383642c683 is not leader of this config: current role FOLLOWER
I20240502 01:58:59.081274 1124 raft_consensus.cc:1261] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Refusing update from remote peer 07c427880bd54215847186eab221fce5: Log matching property violated. Preceding OpId in replica: term: 1 index: 471. Preceding OpId from leader: term: 2 index: 473. (index mismatch)
I20240502 01:58:59.081357 1254 raft_consensus.cc:1261] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Refusing update from remote peer 07c427880bd54215847186eab221fce5: Log matching property violated. Preceding OpId in replica: term: 1 index: 471. Preceding OpId from leader: term: 2 index: 473. (index mismatch)
I20240502 01:58:59.081951 4854 consensus_queue.cc:1035] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 472, Last known committed idx: 471, Time since last communication: 0.000s
I20240502 01:58:59.082115 4779 consensus_queue.cc:1035] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 472, Last known committed idx: 471, Time since last communication: 0.000s
I20240502 01:58:59.089673 4425 mvcc.cc:205] Tried to move back new op lower bound from 7023063609671524352 to 7023063609026170880. Current Snapshot: MvccSnapshot[applied={T|T < 7023063609671524352}]
I20240502 01:58:59.412724 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:59.434195 2347 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:59.469172 1234 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:59.475070 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:59.483881 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:59.498482 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:59.499982 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:59.606885 1254 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "609e71d9a50b4497a0410eb09734c169"
dest_uuid: "4d8c8df1c10b41fab826db1b2ef82e87"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41728
I20240502 01:58:59.607056 1254 raft_consensus.cc:595] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:58:59.609889 3971 raft_consensus.cc:979] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87: : Instructing follower 09401a31cfd046a78a67a2d51e290224 to start an election
I20240502 01:58:59.610118 3971 raft_consensus.cc:1067] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Signalling peer 09401a31cfd046a78a67a2d51e290224 to start an election
W20240502 01:58:59.622706 1215 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33482: Service unavailable: leader transfer in progress
W20240502 01:58:59.626137 1215 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33482: Service unavailable: leader transfer in progress
I20240502 01:58:59.627934 2367 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "609e71d9a50b4497a0410eb09734c169"
dest_uuid: "09401a31cfd046a78a67a2d51e290224"
from {username='slave'} at 127.27.111.194:40639
I20240502 01:58:59.628068 2367 raft_consensus.cc:482] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:58:59.628118 2367 raft_consensus.cc:3037] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:59.629215 2367 raft_consensus.cc:504] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:59.629639 2367 leader_election.cc:288] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [CANDIDATE]: Term 2 election: Requested vote from peers a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473), 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
W20240502 01:58:59.630226 1215 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33482: Service unavailable: leader transfer in progress
I20240502 01:58:59.630578 2364 raft_consensus.cc:1226] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [term 2 FOLLOWER]: Rejecting Update request from peer 4d8c8df1c10b41fab826db1b2ef82e87 for earlier term 1. Current term is 2. Ops: []
I20240502 01:58:59.630630 1254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609e71d9a50b4497a0410eb09734c169" candidate_uuid: "09401a31cfd046a78a67a2d51e290224" candidate_term: 2 candidate_status { last_received { term: 1 index: 482 } } ignore_live_leader: true dest_uuid: "4d8c8df1c10b41fab826db1b2ef82e87"
I20240502 01:58:59.630770 1254 raft_consensus.cc:3032] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:58:59.630812 1254 raft_consensus.cc:729] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Running, Role: LEADER
I20240502 01:58:59.630957 1254 consensus_queue.cc:260] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 482, Committed index: 482, Last appended: 1.482, Last appended by leader: 482, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:59.631062 1254 raft_consensus.cc:3037] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:59.632305 1254 raft_consensus.cc:2454] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09401a31cfd046a78a67a2d51e290224 in term 2.
I20240502 01:58:59.633180 2300 leader_election.cc:302] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09401a31cfd046a78a67a2d51e290224, 4d8c8df1c10b41fab826db1b2ef82e87; no voters:
I20240502 01:58:59.633622 4776 raft_consensus.cc:2781] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:58:59.633703 4776 raft_consensus.cc:686] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [term 2 LEADER]: Becoming Leader. State: Replica: 09401a31cfd046a78a67a2d51e290224, State: Running, Role: LEADER
I20240502 01:58:59.633850 4776 consensus_queue.cc:237] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 481, Committed index: 481, Last appended: 1.482, Last appended by leader: 482, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:58:59.635739 981 catalog_manager.cc:5484] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 reported cstate change: term changed from 1 to 2, leader changed from 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) to 09401a31cfd046a78a67a2d51e290224 (127.27.111.196). New cstate: current_term: 2 leader_uuid: "09401a31cfd046a78a67a2d51e290224" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
W20240502 01:58:59.640877 1215 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33482: Illegal state: replica 4d8c8df1c10b41fab826db1b2ef82e87 is not leader of this config: current role FOLLOWER
I20240502 01:58:59.644027 1124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609e71d9a50b4497a0410eb09734c169" candidate_uuid: "09401a31cfd046a78a67a2d51e290224" candidate_term: 2 candidate_status { last_received { term: 1 index: 482 } } ignore_live_leader: true dest_uuid: "a571e02bbe774f45b80bb4383642c683"
I20240502 01:58:59.644166 1124 raft_consensus.cc:3037] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:58:59.660252 1124 raft_consensus.cc:2454] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09401a31cfd046a78a67a2d51e290224 in term 2.
W20240502 01:58:59.677834 1085 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52982: Illegal state: replica a571e02bbe774f45b80bb4383642c683 is not leader of this config: current role FOLLOWER
I20240502 01:58:59.703282 1125 raft_consensus.cc:1261] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Refusing update from remote peer 09401a31cfd046a78a67a2d51e290224: Log matching property violated. Preceding OpId in replica: term: 1 index: 482. Preceding OpId from leader: term: 2 index: 484. (index mismatch)
I20240502 01:58:59.703678 4907 consensus_queue.cc:1035] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 483, Last known committed idx: 481, Time since last communication: 0.000s
I20240502 01:58:59.707336 1254 raft_consensus.cc:1261] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Refusing update from remote peer 09401a31cfd046a78a67a2d51e290224: Log matching property violated. Preceding OpId in replica: term: 1 index: 482. Preceding OpId from leader: term: 2 index: 484. (index mismatch)
I20240502 01:58:59.712729 4907 consensus_queue.cc:1035] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 483, Last known committed idx: 482, Time since last communication: 0.000s
I20240502 01:58:59.912066 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:59.920114 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:59.927966 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:58:59.929591 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:59.940399 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:58:59.952132 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:59.962872 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:00.268445 2367 consensus_queue.cc:237] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 498, Committed index: 498, Last appended: 2.499, Last appended by leader: 482, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:59:00.270834 1123 raft_consensus.cc:1261] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Refusing update from remote peer 09401a31cfd046a78a67a2d51e290224: Log matching property violated. Preceding OpId in replica: term: 2 index: 498. Preceding OpId from leader: term: 2 index: 500. (index mismatch)
I20240502 01:59:00.271448 4967 consensus_queue.cc:1035] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 500, Last known committed idx: 498, Time since last communication: 0.000s
I20240502 01:59:00.273279 1124 raft_consensus.cc:1203] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.498->[2.499-2.500] Dedup: 2.499->[2.500-2.500]
I20240502 01:59:00.274580 4967 raft_consensus.cc:2932] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [term 2 LEADER]: Committing config change with OpId 2.500: config changed from index 303 to 500, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New config: { opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:59:00.278959 981 catalog_manager.cc:5484] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 reported cstate change: config changed from index 303 to 500, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New cstate: current_term: 2 leader_uuid: "09401a31cfd046a78a67a2d51e290224" committed_config { opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:59:00.283581 1124 raft_consensus.cc:2932] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Committing config change with OpId 2.500: config changed from index 303 to 500, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New config: { opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:59:00.304446 1234 tablet_service.cc:1514] Processing DeleteTablet for tablet 609e71d9a50b4497a0410eb09734c169 with delete_type TABLET_DATA_TOMBSTONED (TS 4d8c8df1c10b41fab826db1b2ef82e87 not found in new config with opid_index 500) from {username='slave'} at 127.0.0.1:40794
I20240502 01:59:00.318500 5005 tablet_replica.cc:330] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87: stopping tablet replica
I20240502 01:59:00.318790 5005 raft_consensus.cc:2229] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:00.318996 5005 pending_rounds.cc:62] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87: Trying to abort 1 pending ops.
I20240502 01:59:00.319123 5005 pending_rounds.cc:68] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87: Aborting op as it isn't in flight: id { term: 2 index: 499 } timestamp: 7023063614531694592 op_type: WRITE_OP write_request { tablet_id: "609e71d9a50b4497a0410eb09734c169" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000j\371Sx\304\263\377\373\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a04d114b4754c6fc" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023063614492991488 authz_token { token_data: "\010\376\350\313\261\006\"3\n\005slave\022*\n a79d488e199a471f817d760819b107cd\020\001\030\001 \001(\001" signature: "<redacted>""\204>U8g\365\311\3011\211\341\240\350\020kW)\252[\203\327fj\010\\]|\227\222\256\201*\017\341n=\210r\261kX\361\221\0025\335\017\t\343\310:\3614\345\213@vO\311\253\t\216\326 " signing_key_seq_num: 0 } } request_id { client_id: "2178659be60b4f62b82caad082a81923" seq_no: 1491 first_incomplete_seq_no: 1491 attempt_no: 0 }
I20240502 01:59:00.319643 5005 raft_consensus.cc:2258] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:00.320819 5005 ts_tablet_manager.cc:1875] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:00.335742 5005 ts_tablet_manager.cc:1888] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.499
I20240502 01:59:00.336102 5005 log.cc:1196] T 609e71d9a50b4497a0410eb09734c169 P 4d8c8df1c10b41fab826db1b2ef82e87: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/609e71d9a50b4497a0410eb09734c169
I20240502 01:59:00.337471 967 catalog_manager.cc:4830] TS 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267): tablet 609e71d9a50b4497a0410eb09734c169 (table rebalance_test_table_0 [id=a79d488e199a471f817d760819b107cd]) successfully deleted
I20240502 01:59:00.520773 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:00.522504 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:00.542835 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:00.574083 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:00.586133 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:00.589248 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:00.674677 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:00.880945 2625 consensus_queue.cc:237] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 639, Committed index: 639, Last appended: 2.639, Last appended by leader: 476, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 640 OBSOLETE_local: false peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:59:00.882488 2364 raft_consensus.cc:1261] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224 [term 2 FOLLOWER]: Refusing update from remote peer 72de8d04b745434f923594c920aff90c: Log matching property violated. Preceding OpId in replica: term: 2 index: 639. Preceding OpId from leader: term: 2 index: 640. (index mismatch)
I20240502 01:59:00.892839 4634 consensus_queue.cc:1035] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 640, Last known committed idx: 639, Time since last communication: 0.000s
I20240502 01:59:00.896162 4790 raft_consensus.cc:2932] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c [term 2 LEADER]: Committing config change with OpId 2.640: config changed from index 410 to 640, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New config: { opid_index: 640 OBSOLETE_local: false peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:59:00.899657 981 catalog_manager.cc:5484] T 90948f77e44241e1883249fbcdcf1515 P 72de8d04b745434f923594c920aff90c reported cstate change: config changed from index 410 to 640, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New cstate: current_term: 2 leader_uuid: "72de8d04b745434f923594c920aff90c" committed_config { opid_index: 640 OBSOLETE_local: false peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:59:00.900403 2367 raft_consensus.cc:2932] T 90948f77e44241e1883249fbcdcf1515 P 09401a31cfd046a78a67a2d51e290224 [term 2 FOLLOWER]: Committing config change with OpId 2.640: config changed from index 410 to 640, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New config: { opid_index: 640 OBSOLETE_local: false peers { permanent_uuid: "72de8d04b745434f923594c920aff90c" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 45287 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:59:00.930411 1103 tablet_service.cc:1514] Processing DeleteTablet for tablet 90948f77e44241e1883249fbcdcf1515 with delete_type TABLET_DATA_TOMBSTONED (TS a571e02bbe774f45b80bb4383642c683 not found in new config with opid_index 640) from {username='slave'} at 127.0.0.1:51778
I20240502 01:59:00.930985 5073 tablet_replica.cc:330] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683: stopping tablet replica
I20240502 01:59:00.931195 5073 raft_consensus.cc:2229] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:00.931340 5073 pending_rounds.cc:62] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683: Trying to abort 1 pending ops.
I20240502 01:59:00.931464 5073 pending_rounds.cc:68] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683: Aborting op as it isn't in flight: id { term: 2 index: 639 } timestamp: 7023063616988323840 op_type: WRITE_OP write_request { tablet_id: "90948f77e44241e1883249fbcdcf1515" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000M\375\375Q\213\341\245T\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""454b9d2e4e787372" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023063616949104640 authz_token { token_data: "\010\376\350\313\261\006\"3\n\005slave\022*\n f35b0e8f48d64f5884fb03539ed2621e\020\001\030\001 \001(\001" signature: "<redacted>""\013\241\326v\035\341\rs1\321\"\234}Z\236a\331\357l\324<\237\375Tp\313\005\243\214nH2)\360*\351\010+\021\207m\'s\344t\304P\007\315(h<\251\0213\312<\351\261\205\363\227}1" signing_key_seq_num: 0 } } request_id { client_id: "ebe5808c163a473bba50dd163b0db452" seq_no: 1843 first_incomplete_seq_no: 1843 attempt_no: 0 }
I20240502 01:59:00.940968 5073 raft_consensus.cc:2258] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:00.962924 5073 ts_tablet_manager.cc:1875] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:00.966590 5073 ts_tablet_manager.cc:1888] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.639
I20240502 01:59:00.966756 5073 log.cc:1196] T 90948f77e44241e1883249fbcdcf1515 P a571e02bbe774f45b80bb4383642c683: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/90948f77e44241e1883249fbcdcf1515
I20240502 01:59:00.967545 968 catalog_manager.cc:4830] TS a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473): tablet 90948f77e44241e1883249fbcdcf1515 (table rebalance_test_table_3 [id=f35b0e8f48d64f5884fb03539ed2621e]) successfully deleted
I20240502 01:59:01.214484 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:01.242048 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:01.260978 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:01.263722 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:01.272581 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:01.278616 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:01.279575 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:01.560631 2366 consensus_queue.cc:237] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 540, Committed index: 540, Last appended: 2.540, Last appended by leader: 446, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 541 OBSOLETE_local: false peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } }
I20240502 01:59:01.561650 2761 raft_consensus.cc:1261] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb [term 2 FOLLOWER]: Refusing update from remote peer 09401a31cfd046a78a67a2d51e290224: Log matching property violated. Preceding OpId in replica: term: 2 index: 540. Preceding OpId from leader: term: 2 index: 541. (index mismatch)
I20240502 01:59:01.562284 4907 consensus_queue.cc:1035] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 541, Last known committed idx: 540, Time since last communication: 0.000s
I20240502 01:59:01.564414 4776 raft_consensus.cc:2932] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 [term 2 LEADER]: Committing config change with OpId 2.541: config changed from index 303 to 541, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New config: { opid_index: 541 OBSOLETE_local: false peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:59:01.564777 2763 raft_consensus.cc:2932] T d0cde29604ac4292be1ead0a69aa6908 P 37279883fafe432f81fe7b8224dbe5fb [term 2 FOLLOWER]: Committing config change with OpId 2.541: config changed from index 303 to 541, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New config: { opid_index: 541 OBSOLETE_local: false peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:59:01.568303 976 catalog_manager.cc:5484] T d0cde29604ac4292be1ead0a69aa6908 P 09401a31cfd046a78a67a2d51e290224 reported cstate change: config changed from index 303 to 541, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New cstate: current_term: 2 leader_uuid: "09401a31cfd046a78a67a2d51e290224" committed_config { opid_index: 541 OBSOLETE_local: false peers { permanent_uuid: "37279883fafe432f81fe7b8224dbe5fb" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 33361 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } }
I20240502 01:59:01.643193 1103 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cde29604ac4292be1ead0a69aa6908 with delete_type TABLET_DATA_TOMBSTONED (TS a571e02bbe774f45b80bb4383642c683 not found in new config with opid_index 541) from {username='slave'} at 127.0.0.1:51778
I20240502 01:59:01.643520 5147 tablet_replica.cc:330] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683: stopping tablet replica
I20240502 01:59:01.643712 5147 raft_consensus.cc:2229] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:01.643900 5147 raft_consensus.cc:2258] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:01.644731 5147 ts_tablet_manager.cc:1875] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:01.651722 5147 ts_tablet_manager.cc:1888] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.540
I20240502 01:59:01.651935 5147 log.cc:1196] T d0cde29604ac4292be1ead0a69aa6908 P a571e02bbe774f45b80bb4383642c683: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/d0cde29604ac4292be1ead0a69aa6908
I20240502 01:59:01.666159 968 catalog_manager.cc:4830] TS a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473): tablet d0cde29604ac4292be1ead0a69aa6908 (table rebalance_test_table_1 [id=e6f3958b51cd48bfb0764c9b91209835]) successfully deleted
I20240502 01:59:01.856169 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:01.884138 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:01.891489 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:01.896903 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:01.903579 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:01.946097 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:02.067773 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:02.239539 2497 consensus_queue.cc:237] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 574, Committed index: 574, Last appended: 2.574, Last appended by leader: 471, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 575 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } }
I20240502 01:59:02.248565 1125 raft_consensus.cc:1261] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Refusing update from remote peer 07c427880bd54215847186eab221fce5: Log matching property violated. Preceding OpId in replica: term: 2 index: 574. Preceding OpId from leader: term: 2 index: 575. (index mismatch)
I20240502 01:59:02.249208 4978 consensus_queue.cc:1035] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 575, Last known committed idx: 573, Time since last communication: 0.000s
I20240502 01:59:02.252152 4978 raft_consensus.cc:2932] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 [term 2 LEADER]: Committing config change with OpId 2.575: config changed from index 319 to 575, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New config: { opid_index: 575 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:59:02.255717 981 catalog_manager.cc:5484] T fa93c869b01b4af69678e378df7db684 P 07c427880bd54215847186eab221fce5 reported cstate change: config changed from index 319 to 575, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New cstate: current_term: 2 leader_uuid: "07c427880bd54215847186eab221fce5" committed_config { opid_index: 575 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:59:02.257388 1123 raft_consensus.cc:2932] T fa93c869b01b4af69678e378df7db684 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Committing config change with OpId 2.575: config changed from index 319 to 575, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New config: { opid_index: 575 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } }
I20240502 01:59:02.307090 1234 tablet_service.cc:1514] Processing DeleteTablet for tablet fa93c869b01b4af69678e378df7db684 with delete_type TABLET_DATA_TOMBSTONED (TS 4d8c8df1c10b41fab826db1b2ef82e87 not found in new config with opid_index 575) from {username='slave'} at 127.0.0.1:40794
I20240502 01:59:02.316372 5220 tablet_replica.cc:330] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87: stopping tablet replica
I20240502 01:59:02.316617 5220 raft_consensus.cc:2229] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:02.316844 5220 raft_consensus.cc:2258] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:02.319344 5220 ts_tablet_manager.cc:1875] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:02.329649 5220 ts_tablet_manager.cc:1888] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.574
I20240502 01:59:02.330157 5220 log.cc:1196] T fa93c869b01b4af69678e378df7db684 P 4d8c8df1c10b41fab826db1b2ef82e87: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/fa93c869b01b4af69678e378df7db684
I20240502 01:59:02.330935 967 catalog_manager.cc:4830] TS 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267): tablet fa93c869b01b4af69678e378df7db684 (table rebalance_test_table_2 [id=2fba41bfdcb948309c7c7ecff39e9c06]) successfully deleted
I20240502 01:59:02.614639 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:02.619704 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:02.690266 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:02.701355 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:02.706761 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:02.761245 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:02.781466 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:03.122076 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:03.149667 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:03.151999 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:03.159463 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:03.177012 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:03.183027 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:03.222954 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:03.923825 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:03.937137 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:04.018380 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:04.034507 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:04.040757 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:04.043220 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:04.051362 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:04.490537 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:04.501899 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:04.514597 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:04.522712 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:04.529078 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:04.567041 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:04.618291 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:04.836169 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:04.837149 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:04.839130 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:04.839354 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:04.839668 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:04.840204 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:04.863201 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
8cfd9cbfd05e40b09dbec2dce9761c79 | 127.27.111.254:38251 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.27.111.212:36285 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
07c427880bd54215847186eab221fce5 | 127.27.111.197:39309 | HEALTHY | <none> | 2 | 0
09401a31cfd046a78a67a2d51e290224 | 127.27.111.196:43819 | HEALTHY | <none> | 2 | 0
37279883fafe432f81fe7b8224dbe5fb | 127.27.111.199:33361 | HEALTHY | <none> | 0 | 0
3f073f261d9447b99220aa532c2b1072 | 127.27.111.195:45227 | HEALTHY | <none> | 3 | 0
4d8c8df1c10b41fab826db1b2ef82e87 | 127.27.111.194:33267 | HEALTHY | <none> | 5 | 0
72de8d04b745434f923594c920aff90c | 127.27.111.198:45287 | HEALTHY | <none> | 2 | 0
a571e02bbe774f45b80bb4383642c683 | 127.27.111.193:39473 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.27.111.193 | experimental | 127.27.111.193:39473
local_ip_for_outbound_sockets | 127.27.111.194 | experimental | 127.27.111.194:33267
local_ip_for_outbound_sockets | 127.27.111.195 | experimental | 127.27.111.195:45227
local_ip_for_outbound_sockets | 127.27.111.196 | experimental | 127.27.111.196:43819
local_ip_for_outbound_sockets | 127.27.111.197 | experimental | 127.27.111.197:39309
local_ip_for_outbound_sockets | 127.27.111.198 | experimental | 127.27.111.198:45287
local_ip_for_outbound_sockets | 127.27.111.199 | experimental | 127.27.111.199:33361
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.27.111.193:39473
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.27.111.194:33267
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.27.111.195:45227
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.27.111.196:43819
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.27.111.197:39309
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.27.111.198:45287
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.27.111.199:33361
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.27.111.212:36285 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 01:59:04.888376 28095 log_verifier.cc:126] Checking tablet 0fa2250275ba4eb391f88f9ee0fd52e9
I20240502 01:59:04.905684 28095 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 0fa2250275ba4eb391f88f9ee0fd52e9
I20240502 01:59:04.905779 28095 log_verifier.cc:126] Checking tablet 299019dc3d0848ad9324a84e319b9222
I20240502 01:59:05.021443 28095 log_verifier.cc:177] Verified matching terms for 637 ops in tablet 299019dc3d0848ad9324a84e319b9222
I20240502 01:59:05.021756 28095 log_verifier.cc:126] Checking tablet 5b90b275dd77400f9aaa1ffc0ea4be18
I20240502 01:59:05.114810 28095 log_verifier.cc:177] Verified matching terms for 575 ops in tablet 5b90b275dd77400f9aaa1ffc0ea4be18
I20240502 01:59:05.115015 28095 log_verifier.cc:126] Checking tablet 5bba25f4abe945429b0cc8ac93e12c8a
I20240502 01:59:05.224970 28095 log_verifier.cc:177] Verified matching terms for 691 ops in tablet 5bba25f4abe945429b0cc8ac93e12c8a
I20240502 01:59:05.225162 28095 log_verifier.cc:126] Checking tablet 609e71d9a50b4497a0410eb09734c169
I20240502 01:59:05.326532 28095 log_verifier.cc:177] Verified matching terms for 630 ops in tablet 609e71d9a50b4497a0410eb09734c169
I20240502 01:59:05.326781 28095 log_verifier.cc:126] Checking tablet 690f9ec1b4c34a919a9a09ae7c618ef3
I20240502 01:59:05.419503 28095 log_verifier.cc:177] Verified matching terms for 551 ops in tablet 690f9ec1b4c34a919a9a09ae7c618ef3
I20240502 01:59:05.419679 28095 log_verifier.cc:126] Checking tablet 82337220b48d45fba654828a31eabbf4
I20240502 01:59:05.520479 28095 log_verifier.cc:177] Verified matching terms for 634 ops in tablet 82337220b48d45fba654828a31eabbf4
I20240502 01:59:05.520700 28095 log_verifier.cc:126] Checking tablet 844edd610a734b0994f59a4b0e29af0b
I20240502 01:59:05.633760 28095 log_verifier.cc:177] Verified matching terms for 666 ops in tablet 844edd610a734b0994f59a4b0e29af0b
I20240502 01:59:05.634060 28095 log_verifier.cc:126] Checking tablet 90948f77e44241e1883249fbcdcf1515
I20240502 01:59:05.786201 28095 log_verifier.cc:177] Verified matching terms for 757 ops in tablet 90948f77e44241e1883249fbcdcf1515
I20240502 01:59:05.786569 28095 log_verifier.cc:126] Checking tablet a59a5084412f4082bdd35e7c546e71b4
I20240502 01:59:05.887727 28095 log_verifier.cc:177] Verified matching terms for 557 ops in tablet a59a5084412f4082bdd35e7c546e71b4
I20240502 01:59:05.888006 28095 log_verifier.cc:126] Checking tablet a6da3730570e48d2aba7457e9d41dfcc
I20240502 01:59:06.007416 28095 log_verifier.cc:177] Verified matching terms for 726 ops in tablet a6da3730570e48d2aba7457e9d41dfcc
I20240502 01:59:06.007665 28095 log_verifier.cc:126] Checking tablet c0e16afa25404fa78e03547f7b140283
I20240502 01:59:06.114820 28095 log_verifier.cc:177] Verified matching terms for 664 ops in tablet c0e16afa25404fa78e03547f7b140283
I20240502 01:59:06.115046 28095 log_verifier.cc:126] Checking tablet c8a946d0b927414f88c615d54ce5fc7f
I20240502 01:59:06.216900 28095 log_verifier.cc:177] Verified matching terms for 627 ops in tablet c8a946d0b927414f88c615d54ce5fc7f
I20240502 01:59:06.217136 28095 log_verifier.cc:126] Checking tablet d0cde29604ac4292be1ead0a69aa6908
I20240502 01:59:06.318280 28095 log_verifier.cc:177] Verified matching terms for 635 ops in tablet d0cde29604ac4292be1ead0a69aa6908
I20240502 01:59:06.318547 28095 log_verifier.cc:126] Checking tablet dd52517cbc5b43b798af83233b3759b9
I20240502 01:59:06.445941 28095 log_verifier.cc:177] Verified matching terms for 749 ops in tablet dd52517cbc5b43b798af83233b3759b9
I20240502 01:59:06.446230 28095 log_verifier.cc:126] Checking tablet fa93c869b01b4af69678e378df7db684
I20240502 01:59:06.552471 28095 log_verifier.cc:177] Verified matching terms for 654 ops in tablet fa93c869b01b4af69678e378df7db684
I20240502 01:59:06.554800 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.200:0
--local_ip_for_outbound_sockets=127.27.111.200
--webserver_interface=127.27.111.200
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--builtin_ntp_servers=127.27.111.212:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:59:06.675746 5454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:06.675990 5454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:59:06.676033 5454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:06.679217 5454 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:59:06.679301 5454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:59:06.679385 5454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.200
I20240502 01:59:06.683259 5454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.111.200
--webserver_port=0
--tserver_master_addrs=127.27.111.254:38251
--never_fsync=true
--heap_profile_path=/tmp/kudu.5454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.200
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:59:06.684221 5454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:59:06.685402 5454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:06.694370 5460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:06.694928 5459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:06.695605 5454 server_base.cc:1034] running on GCE node
W20240502 01:59:06.696309 5462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:06.696753 5454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:59:06.697394 5454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:59:06.698544 5454 hybrid_clock.cc:648] HybridClock initialized: now 1714615146698533 us; error 30 us; skew 500 ppm
I20240502 01:59:06.700515 5454 webserver.cc:460] Webserver started at http://127.27.111.200:45837/ using document root <none> and password file <none>
I20240502 01:59:06.701113 5454 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:06.701189 5454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:06.701387 5454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:06.703218 5454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "51a574cb98a84e3aabf09db9a952b712"
format_stamp: "Formatted at 2024-05-02 01:59:06 on dist-test-slave-0pw1"
I20240502 01:59:06.703658 5454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "51a574cb98a84e3aabf09db9a952b712"
format_stamp: "Formatted at 2024-05-02 01:59:06 on dist-test-slave-0pw1"
I20240502 01:59:06.707134 5454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:59:06.709338 5468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:06.710620 5454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:59:06.710738 5454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "51a574cb98a84e3aabf09db9a952b712"
format_stamp: "Formatted at 2024-05-02 01:59:06 on dist-test-slave-0pw1"
I20240502 01:59:06.710851 5454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:06.732190 5454 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:06.732874 5454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:59:06.733031 5454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:06.733557 5454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:06.734508 5454 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:06.734558 5454 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:06.734627 5454 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:06.734652 5454 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:06.743575 5454 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.200:35643
I20240502 01:59:06.743702 5581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.200:35643 every 8 connection(s)
I20240502 01:59:06.744617 5454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:59:06.751914 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 5454
I20240502 01:59:06.752063 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:59:06.754331 5582 heartbeater.cc:347] Connected to a master server at 127.27.111.254:38251
I20240502 01:59:06.754618 5582 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:06.755100 5582 heartbeater.cc:511] Master 127.27.111.254:38251 requested a full tablet report, sending...
I20240502 01:59:06.756152 974 ts_manager.cc:194] Registered new tserver with Master: 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200:35643)
I20240502 01:59:06.757012 974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.200:55741
I20240502 01:59:06.957170 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:06.959672 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:06.970124 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:06.977612 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:06.977953 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:06.985239 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:06.996089 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.007412 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.096851 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:07.103343 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.103439 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:07.103591 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.106714 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:07.110821 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.119632 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:07.125272 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.182191 1124 consensus_queue.cc:237] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 551, Committed index: 551, Last appended: 1.551, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:07.183799 2366 raft_consensus.cc:1261] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 1 index: 551. Preceding OpId from leader: term: 1 index: 552. (index mismatch)
I20240502 01:59:07.184281 5690 consensus_queue.cc:1035] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 552, Last known committed idx: 551, Time since last communication: 0.000s
I20240502 01:59:07.186139 2853 raft_consensus.cc:2932] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Committing config change with OpId 1.552: config changed from index 159 to 552, VOTER 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200) added. New config: { opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:07.186574 2366 raft_consensus.cc:2932] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Committing config change with OpId 1.552: config changed from index 159 to 552, VOTER 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200) added. New config: { opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:07.188797 980 catalog_manager.cc:5484] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224 reported cstate change: config changed from index 159 to 552, VOTER 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
W20240502 01:59:07.190244 1061 consensus_peers.cc:485] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 -> Peer 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200:35643): Couldn't send request to peer 51a574cb98a84e3aabf09db9a952b712. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 690f9ec1b4c34a919a9a09ae7c618ef3. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:07.208711 1124 consensus_queue.cc:237] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 691, Committed index: 691, Last appended: 1.691, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 692 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:07.210296 1383 raft_consensus.cc:1261] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 1 index: 691. Preceding OpId from leader: term: 1 index: 692. (index mismatch)
W20240502 01:59:07.210496 1061 consensus_peers.cc:485] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 -> Peer 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200:35643): Couldn't send request to peer 51a574cb98a84e3aabf09db9a952b712. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bba25f4abe945429b0cc8ac93e12c8a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:07.210775 5690 consensus_queue.cc:1035] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 692, Last known committed idx: 691, Time since last communication: 0.000s
I20240502 01:59:07.212452 2853 raft_consensus.cc:2932] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Committing config change with OpId 1.692: config changed from index -1 to 692, VOTER 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200) added. New config: { opid_index: 692 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:07.213984 1383 raft_consensus.cc:2932] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Committing config change with OpId 1.692: config changed from index -1 to 692, VOTER 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200) added. New config: { opid_index: 692 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:07.215097 982 catalog_manager.cc:5484] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 reported cstate change: config changed from index -1 to 692, VOTER 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: 692 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:07.235283 1254 consensus_queue.cc:237] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 749, Committed index: 749, Last appended: 1.749, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 750 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:07.237035 2496 raft_consensus.cc:1261] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Refusing update from remote peer 4d8c8df1c10b41fab826db1b2ef82e87: Log matching property violated. Preceding OpId in replica: term: 1 index: 749. Preceding OpId from leader: term: 1 index: 750. (index mismatch)
I20240502 01:59:07.237434 5731 consensus_queue.cc:1035] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 750, Last known committed idx: 749, Time since last communication: 0.000s
I20240502 01:59:07.239296 5731 raft_consensus.cc:2932] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Committing config change with OpId 1.750: config changed from index 271 to 750, VOTER 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200) added. New config: { opid_index: 750 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:07.239725 2496 raft_consensus.cc:2932] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Committing config change with OpId 1.750: config changed from index 271 to 750, VOTER 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200) added. New config: { opid_index: 750 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
W20240502 01:59:07.240033 1192 consensus_peers.cc:485] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 -> Peer 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200:35643): Couldn't send request to peer 51a574cb98a84e3aabf09db9a952b712. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd52517cbc5b43b798af83233b3759b9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:07.242039 976 catalog_manager.cc:5484] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 reported cstate change: config changed from index 271 to 750, VOTER 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" committed_config { opid_index: 750 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:07.260553 2366 consensus_queue.cc:237] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 630, Committed index: 630, Last appended: 2.630, Last appended by leader: 482, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 631 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:07.261991 1124 raft_consensus.cc:1261] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Refusing update from remote peer 09401a31cfd046a78a67a2d51e290224: Log matching property violated. Preceding OpId in replica: term: 2 index: 630. Preceding OpId from leader: term: 2 index: 631. (index mismatch)
I20240502 01:59:07.262476 5634 consensus_queue.cc:1035] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 631, Last known committed idx: 630, Time since last communication: 0.000s
I20240502 01:59:07.264335 5633 raft_consensus.cc:2932] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [term 2 LEADER]: Committing config change with OpId 2.631: config changed from index 500 to 631, VOTER 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200) added. New config: { opid_index: 631 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:07.264708 1124 raft_consensus.cc:2932] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Committing config change with OpId 2.631: config changed from index 500 to 631, VOTER 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200) added. New config: { opid_index: 631 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:07.266885 982 catalog_manager.cc:5484] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 reported cstate change: config changed from index 500 to 631, VOTER 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200) added. New cstate: current_term: 2 leader_uuid: "09401a31cfd046a78a67a2d51e290224" committed_config { opid_index: 631 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
W20240502 01:59:07.267359 2303 consensus_peers.cc:485] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 -> Peer 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200:35643): Couldn't send request to peer 51a574cb98a84e3aabf09db9a952b712. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 609e71d9a50b4497a0410eb09734c169. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:07.328128 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:07.329465 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:07.335845 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.337965 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.339812 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:07.342576 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.348251 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.359449 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:07.448694 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.451771 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:07.456382 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.457165 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:07.459132 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:07.461226 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.463306 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:07.473881 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.573375 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.573884 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.578779 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:07.579061 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.579319 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:07.593448 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:07.594177 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.606335 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:07.691524 5889 ts_tablet_manager.cc:902] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712: Initiating tablet copy from peer a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:59:07.692376 5889 tablet_copy_client.cc:308] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:39473
I20240502 01:59:07.697994 1144 tablet_copy_service.cc:140] P a571e02bbe774f45b80bb4383642c683: Received BeginTabletCopySession request for tablet 5bba25f4abe945429b0cc8ac93e12c8a from peer 51a574cb98a84e3aabf09db9a952b712 ({username='slave'} at 127.27.111.200:49239)
I20240502 01:59:07.698138 1144 tablet_copy_service.cc:161] P a571e02bbe774f45b80bb4383642c683: Beginning new tablet copy session on tablet 5bba25f4abe945429b0cc8ac93e12c8a from peer 51a574cb98a84e3aabf09db9a952b712 at {username='slave'} at 127.27.111.200:49239: session id = 51a574cb98a84e3aabf09db9a952b712-5bba25f4abe945429b0cc8ac93e12c8a
I20240502 01:59:07.699288 1144 tablet_copy_source_session.cc:205] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:07.704236 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.707060 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:07.707494 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:07.708518 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.713095 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.721040 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:07.722496 5889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bba25f4abe945429b0cc8ac93e12c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.729966 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:07.740142 5907 ts_tablet_manager.cc:902] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712: Initiating tablet copy from peer 09401a31cfd046a78a67a2d51e290224 (127.27.111.196:43819)
I20240502 01:59:07.740696 5907 tablet_copy_client.cc:308] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:43819
I20240502 01:59:07.744964 5889 tablet_copy_client.cc:783] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:07.745285 5889 tablet_copy_client.cc:647] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:07.746727 2387 tablet_copy_service.cc:140] P 09401a31cfd046a78a67a2d51e290224: Received BeginTabletCopySession request for tablet 609e71d9a50b4497a0410eb09734c169 from peer 51a574cb98a84e3aabf09db9a952b712 ({username='slave'} at 127.27.111.200:36411)
I20240502 01:59:07.746876 2387 tablet_copy_service.cc:161] P 09401a31cfd046a78a67a2d51e290224: Beginning new tablet copy session on tablet 609e71d9a50b4497a0410eb09734c169 from peer 51a574cb98a84e3aabf09db9a952b712 at {username='slave'} at 127.27.111.200:36411: session id = 51a574cb98a84e3aabf09db9a952b712-609e71d9a50b4497a0410eb09734c169
I20240502 01:59:07.748211 2387 tablet_copy_source_session.cc:205] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:07.749375 5907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609e71d9a50b4497a0410eb09734c169. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.752072 5907 tablet_copy_client.cc:783] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:07.752279 5907 tablet_copy_client.cc:647] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:07.755836 5912 ts_tablet_manager.cc:902] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712: Initiating tablet copy from peer a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473)
I20240502 01:59:07.756444 5912 tablet_copy_client.cc:308] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:39473
I20240502 01:59:07.756919 1144 tablet_copy_service.cc:140] P a571e02bbe774f45b80bb4383642c683: Received BeginTabletCopySession request for tablet 690f9ec1b4c34a919a9a09ae7c618ef3 from peer 51a574cb98a84e3aabf09db9a952b712 ({username='slave'} at 127.27.111.200:49239)
I20240502 01:59:07.757040 1144 tablet_copy_service.cc:161] P a571e02bbe774f45b80bb4383642c683: Beginning new tablet copy session on tablet 690f9ec1b4c34a919a9a09ae7c618ef3 from peer 51a574cb98a84e3aabf09db9a952b712 at {username='slave'} at 127.27.111.200:49239: session id = 51a574cb98a84e3aabf09db9a952b712-690f9ec1b4c34a919a9a09ae7c618ef3
I20240502 01:59:07.757853 1144 tablet_copy_source_session.cc:205] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:07.758867 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:07.759279 5912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690f9ec1b4c34a919a9a09ae7c618ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.761250 5912 tablet_copy_client.cc:783] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:07.761408 5912 tablet_copy_client.cc:647] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:07.761837 5582 heartbeater.cc:503] Master 127.27.111.254:38251 was elected leader, sending a full tablet report...
I20240502 01:59:07.762377 5889 tablet_copy_client.cc:523] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:07.763489 5912 tablet_copy_client.cc:523] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:07.764324 5889 tablet_bootstrap.cc:492] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712: Bootstrap starting.
I20240502 01:59:07.764600 5912 tablet_bootstrap.cc:492] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712: Bootstrap starting.
I20240502 01:59:07.767045 5907 tablet_copy_client.cc:523] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:07.768461 5907 tablet_bootstrap.cc:492] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712: Bootstrap starting.
I20240502 01:59:07.794361 5917 ts_tablet_manager.cc:902] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712: Initiating tablet copy from peer 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267)
I20240502 01:59:07.810281 5917 tablet_copy_client.cc:308] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:33267
I20240502 01:59:07.815222 1276 tablet_copy_service.cc:140] P 4d8c8df1c10b41fab826db1b2ef82e87: Received BeginTabletCopySession request for tablet dd52517cbc5b43b798af83233b3759b9 from peer 51a574cb98a84e3aabf09db9a952b712 ({username='slave'} at 127.27.111.200:35777)
I20240502 01:59:07.815363 1276 tablet_copy_service.cc:161] P 4d8c8df1c10b41fab826db1b2ef82e87: Beginning new tablet copy session on tablet dd52517cbc5b43b798af83233b3759b9 from peer 51a574cb98a84e3aabf09db9a952b712 at {username='slave'} at 127.27.111.200:35777: session id = 51a574cb98a84e3aabf09db9a952b712-dd52517cbc5b43b798af83233b3759b9
I20240502 01:59:07.816321 1276 tablet_copy_source_session.cc:205] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:07.817322 5917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd52517cbc5b43b798af83233b3759b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:07.819824 5917 tablet_copy_client.cc:783] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:07.819999 5917 tablet_copy_client.cc:647] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:07.823395 5917 tablet_copy_client.cc:523] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:07.823921 5907 log.cc:824] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712: Log is configured to *not* fsync() on all Append() calls
I20240502 01:59:07.824657 5917 tablet_bootstrap.cc:492] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712: Bootstrap starting.
I20240502 01:59:07.991680 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.002544 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:08.006803 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.021329 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:08.036777 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:08.068002 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:08.081565 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.082625 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.294315 5912 tablet_bootstrap.cc:492] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712: Bootstrap replayed 1/1 log segments. Stats: ops{read=552 overwritten=0 applied=552 ignored=0} inserts{seen=1222 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:08.294930 5912 tablet_bootstrap.cc:492] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712: Bootstrap complete.
I20240502 01:59:08.295428 5912 ts_tablet_manager.cc:1367] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712: Time spent bootstrapping tablet: real 0.531s user 0.386s sys 0.022s
I20240502 01:59:08.298331 5912 raft_consensus.cc:348] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:08.298766 5912 raft_consensus.cc:729] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a574cb98a84e3aabf09db9a952b712, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.299361 5912 consensus_queue.cc:260] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 552, Last appended: 1.552, Last appended by leader: 552, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:08.300420 5912 ts_tablet_manager.cc:1398] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:59:08.302937 1144 tablet_copy_service.cc:342] P a571e02bbe774f45b80bb4383642c683: Request end of tablet copy session 51a574cb98a84e3aabf09db9a952b712-690f9ec1b4c34a919a9a09ae7c618ef3 received from {username='slave'} at 127.27.111.200:49239
I20240502 01:59:08.303045 1144 tablet_copy_service.cc:433] P a571e02bbe774f45b80bb4383642c683: ending tablet copy session 51a574cb98a84e3aabf09db9a952b712-690f9ec1b4c34a919a9a09ae7c618ef3 on tablet 690f9ec1b4c34a919a9a09ae7c618ef3 with peer 51a574cb98a84e3aabf09db9a952b712
I20240502 01:59:08.309109 5536 raft_consensus.cc:1203] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.551->[1.552-1.552] Dedup: 1.552->[]
I20240502 01:59:08.344976 5907 tablet_bootstrap.cc:492] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712: Bootstrap replayed 1/1 log segments. Stats: ops{read=631 overwritten=0 applied=631 ignored=0} inserts{seen=1300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:08.345319 5907 tablet_bootstrap.cc:492] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712: Bootstrap complete.
I20240502 01:59:08.345646 5907 ts_tablet_manager.cc:1367] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712: Time spent bootstrapping tablet: real 0.577s user 0.415s sys 0.040s
I20240502 01:59:08.346267 5907 raft_consensus.cc:348] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 631 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:08.346405 5907 raft_consensus.cc:729] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a574cb98a84e3aabf09db9a952b712, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.346576 5907 consensus_queue.cc:260] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 631, Last appended: 2.631, Last appended by leader: 631, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 631 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:08.347101 5907 ts_tablet_manager.cc:1398] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:08.347831 2387 tablet_copy_service.cc:342] P 09401a31cfd046a78a67a2d51e290224: Request end of tablet copy session 51a574cb98a84e3aabf09db9a952b712-609e71d9a50b4497a0410eb09734c169 received from {username='slave'} at 127.27.111.200:36411
I20240502 01:59:08.347951 2387 tablet_copy_service.cc:433] P 09401a31cfd046a78a67a2d51e290224: ending tablet copy session 51a574cb98a84e3aabf09db9a952b712-609e71d9a50b4497a0410eb09734c169 on tablet 609e71d9a50b4497a0410eb09734c169 with peer 51a574cb98a84e3aabf09db9a952b712
I20240502 01:59:08.402098 5889 tablet_bootstrap.cc:492] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712: Bootstrap replayed 1/1 log segments. Stats: ops{read=692 overwritten=0 applied=692 ignored=0} inserts{seen=1377 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:08.402647 5889 tablet_bootstrap.cc:492] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712: Bootstrap complete.
I20240502 01:59:08.402979 5889 ts_tablet_manager.cc:1367] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712: Time spent bootstrapping tablet: real 0.639s user 0.471s sys 0.033s
I20240502 01:59:08.403625 5889 raft_consensus.cc:348] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 692 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:08.403788 5889 raft_consensus.cc:729] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a574cb98a84e3aabf09db9a952b712, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.403972 5889 consensus_queue.cc:260] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 692, Last appended: 1.692, Last appended by leader: 692, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 692 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "3f073f261d9447b99220aa532c2b1072" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 45227 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:08.404388 5889 ts_tablet_manager.cc:1398] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:08.405016 1144 tablet_copy_service.cc:342] P a571e02bbe774f45b80bb4383642c683: Request end of tablet copy session 51a574cb98a84e3aabf09db9a952b712-5bba25f4abe945429b0cc8ac93e12c8a received from {username='slave'} at 127.27.111.200:49239
I20240502 01:59:08.405117 1144 tablet_copy_service.cc:433] P a571e02bbe774f45b80bb4383642c683: ending tablet copy session 51a574cb98a84e3aabf09db9a952b712-5bba25f4abe945429b0cc8ac93e12c8a on tablet 5bba25f4abe945429b0cc8ac93e12c8a with peer 51a574cb98a84e3aabf09db9a952b712
I20240502 01:59:08.418825 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.421451 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:08.431836 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:08.431906 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:08.433602 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.441522 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:08.443410 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.443738 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.544088 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:08.544607 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:08.548404 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.557904 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.558660 5917 tablet_bootstrap.cc:492] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712: Bootstrap replayed 1/1 log segments. Stats: ops{read=750 overwritten=0 applied=750 ignored=0} inserts{seen=1420 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:08.558980 5917 tablet_bootstrap.cc:492] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712: Bootstrap complete.
I20240502 01:59:08.559293 5917 ts_tablet_manager.cc:1367] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712: Time spent bootstrapping tablet: real 0.735s user 0.483s sys 0.055s
I20240502 01:59:08.559693 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:08.559973 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:08.560185 5917 raft_consensus.cc:348] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 750 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:08.560371 5917 raft_consensus.cc:729] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a574cb98a84e3aabf09db9a952b712, State: Initialized, Role: FOLLOWER
I20240502 01:59:08.560570 5917 consensus_queue.cc:260] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 750, Last appended: 1.750, Last appended by leader: 750, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 750 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:08.561435 5917 ts_tablet_manager.cc:1398] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:08.565229 1276 tablet_copy_service.cc:342] P 4d8c8df1c10b41fab826db1b2ef82e87: Request end of tablet copy session 51a574cb98a84e3aabf09db9a952b712-dd52517cbc5b43b798af83233b3759b9 received from {username='slave'} at 127.27.111.200:35777
I20240502 01:59:08.565321 1276 tablet_copy_service.cc:433] P 4d8c8df1c10b41fab826db1b2ef82e87: ending tablet copy session 51a574cb98a84e3aabf09db9a952b712-dd52517cbc5b43b798af83233b3759b9 on tablet dd52517cbc5b43b798af83233b3759b9 with peer 51a574cb98a84e3aabf09db9a952b712
I20240502 01:59:08.570916 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.592154 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.647230 5536 raft_consensus.cc:1203] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.749->[1.750-1.750] Dedup: 1.750->[]
I20240502 01:59:08.669605 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.671779 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:08.675668 5536 raft_consensus.cc:1203] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.691->[1.692-1.692] Dedup: 1.692->[]
I20240502 01:59:08.676409 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:08.677031 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:08.688339 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.690788 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.692692 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.703388 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:08.796797 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.801342 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.804723 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:08.807129 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.807932 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.815482 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:08.816727 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:08.817355 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:08.853380 5536 raft_consensus.cc:1203] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.630->[2.631-2.631] Dedup: 2.631->[]
I20240502 01:59:08.873205 1124 consensus_queue.cc:237] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 552, Committed index: 552, Last appended: 1.552, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 553 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:08.874543 5536 raft_consensus.cc:1261] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 1 index: 552. Preceding OpId from leader: term: 1 index: 553. (index mismatch)
I20240502 01:59:08.874987 5690 consensus_queue.cc:1035] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 553, Last known committed idx: 552, Time since last communication: 0.000s
I20240502 01:59:08.877604 6134 raft_consensus.cc:2932] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Committing config change with OpId 1.553: config changed from index 552 to 553, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) evicted. New config: { opid_index: 553 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:08.878150 5536 raft_consensus.cc:2932] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Committing config change with OpId 1.553: config changed from index 552 to 553, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) evicted. New config: { opid_index: 553 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:08.880414 981 catalog_manager.cc:5484] T 690f9ec1b4c34a919a9a09ae7c618ef3 P a571e02bbe774f45b80bb4383642c683 reported cstate change: config changed from index 552 to 553, VOTER 09401a31cfd046a78a67a2d51e290224 (127.27.111.196) evicted. New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: 553 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:08.889186 2347 tablet_service.cc:1514] Processing DeleteTablet for tablet 690f9ec1b4c34a919a9a09ae7c618ef3 with delete_type TABLET_DATA_TOMBSTONED (TS 09401a31cfd046a78a67a2d51e290224 not found in new config with opid_index 553) from {username='slave'} at 127.0.0.1:48676
I20240502 01:59:08.889653 6151 tablet_replica.cc:330] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: stopping tablet replica
I20240502 01:59:08.889947 6151 raft_consensus.cc:2229] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:08.890144 6151 raft_consensus.cc:2258] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:08.891912 6151 ts_tablet_manager.cc:1875] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:08.894127 6151 ts_tablet_manager.cc:1888] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.552
I20240502 01:59:08.894219 6151 log.cc:1196] T 690f9ec1b4c34a919a9a09ae7c618ef3 P 09401a31cfd046a78a67a2d51e290224: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/690f9ec1b4c34a919a9a09ae7c618ef3
I20240502 01:59:08.894897 968 catalog_manager.cc:4830] TS 09401a31cfd046a78a67a2d51e290224 (127.27.111.196:43819): tablet 690f9ec1b4c34a919a9a09ae7c618ef3 (table rebalance_test_table_4 [id=ebb17b8513b14eb0b4a24e8de3c351bf]) successfully deleted
I20240502 01:59:08.942696 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:08.947211 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.952538 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.954515 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.955809 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:08.964452 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:08.972784 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:08.977242 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:09.027756 1124 consensus_queue.cc:237] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 692, Committed index: 692, Last appended: 1.692, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 693 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:09.028986 5536 raft_consensus.cc:1261] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Refusing update from remote peer a571e02bbe774f45b80bb4383642c683: Log matching property violated. Preceding OpId in replica: term: 1 index: 692. Preceding OpId from leader: term: 1 index: 693. (index mismatch)
I20240502 01:59:09.029479 6134 consensus_queue.cc:1035] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 693, Last known committed idx: 692, Time since last communication: 0.000s
I20240502 01:59:09.031355 5690 raft_consensus.cc:2932] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 [term 1 LEADER]: Committing config change with OpId 1.693: config changed from index 692 to 693, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New config: { opid_index: 693 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:09.031769 5536 raft_consensus.cc:2932] T 5bba25f4abe945429b0cc8ac93e12c8a P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Committing config change with OpId 1.693: config changed from index 692 to 693, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New config: { opid_index: 693 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:09.034029 981 catalog_manager.cc:5484] T 5bba25f4abe945429b0cc8ac93e12c8a P a571e02bbe774f45b80bb4383642c683 reported cstate change: config changed from index 692 to 693, VOTER 3f073f261d9447b99220aa532c2b1072 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "a571e02bbe774f45b80bb4383642c683" committed_config { opid_index: 693 OBSOLETE_local: false peers { permanent_uuid: "a571e02bbe774f45b80bb4383642c683" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 39473 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:09.036670 1366 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bba25f4abe945429b0cc8ac93e12c8a with delete_type TABLET_DATA_TOMBSTONED (TS 3f073f261d9447b99220aa532c2b1072 not found in new config with opid_index 693) from {username='slave'} at 127.0.0.1:50588
I20240502 01:59:09.037045 6196 tablet_replica.cc:330] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072: stopping tablet replica
I20240502 01:59:09.037175 6196 raft_consensus.cc:2229] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:09.037295 6196 raft_consensus.cc:2258] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:09.038579 6196 ts_tablet_manager.cc:1875] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:09.040459 6196 ts_tablet_manager.cc:1888] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.692
I20240502 01:59:09.040531 6196 log.cc:1196] T 5bba25f4abe945429b0cc8ac93e12c8a P 3f073f261d9447b99220aa532c2b1072: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/5bba25f4abe945429b0cc8ac93e12c8a
I20240502 01:59:09.041131 968 catalog_manager.cc:4830] TS 3f073f261d9447b99220aa532c2b1072 (127.27.111.195:45227): tablet 5bba25f4abe945429b0cc8ac93e12c8a (table rebalance_test_table_1 [id=e6f3958b51cd48bfb0764c9b91209835]) successfully deleted
I20240502 01:59:09.295127 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.295670 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.301376 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.304940 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:09.305497 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:09.307241 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:09.320281 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:09.327953 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.369391 1254 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dd52517cbc5b43b798af83233b3759b9"
dest_uuid: "4d8c8df1c10b41fab826db1b2ef82e87"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33280
I20240502 01:59:09.369553 1254 raft_consensus.cc:595] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:59:09.424707 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.429797 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:09.430992 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.431125 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:09.435585 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.436928 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:09.455662 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.456925 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:09.547384 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.547825 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.552206 1234 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:09.554620 5516 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:09.560079 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:09.571987 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.586092 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:09.592314 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.603560 2929 raft_consensus.cc:979] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87: : Instructing follower 51a574cb98a84e3aabf09db9a952b712 to start an election
I20240502 01:59:09.603675 2929 raft_consensus.cc:1067] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Signalling peer 51a574cb98a84e3aabf09db9a952b712 to start an election
I20240502 01:59:09.604137 5536 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd52517cbc5b43b798af83233b3759b9"
dest_uuid: "51a574cb98a84e3aabf09db9a952b712"
from {username='slave'} at 127.27.111.194:58749
I20240502 01:59:09.604326 5536 raft_consensus.cc:482] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:09.604439 5536 raft_consensus.cc:3037] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:09.605302 5536 raft_consensus.cc:504] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 750 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:09.606458 5536 leader_election.cc:288] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [CANDIDATE]: Term 2 election: Requested vote from peers 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267), 07c427880bd54215847186eab221fce5 (127.27.111.197:39309)
I20240502 01:59:09.608110 1254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd52517cbc5b43b798af83233b3759b9" candidate_uuid: "51a574cb98a84e3aabf09db9a952b712" candidate_term: 2 candidate_status { last_received { term: 1 index: 750 } } ignore_live_leader: true dest_uuid: "4d8c8df1c10b41fab826db1b2ef82e87"
I20240502 01:59:09.608253 1254 raft_consensus.cc:3032] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:09.608299 1254 raft_consensus.cc:729] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d8c8df1c10b41fab826db1b2ef82e87, State: Running, Role: LEADER
I20240502 01:59:09.608456 1254 consensus_queue.cc:260] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 750, Committed index: 750, Last appended: 1.750, Last appended by leader: 750, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 750 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:09.608637 1254 raft_consensus.cc:3037] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:09.609889 1254 raft_consensus.cc:2454] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a574cb98a84e3aabf09db9a952b712 in term 2.
I20240502 01:59:09.610321 5469 leader_election.cc:302] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d8c8df1c10b41fab826db1b2ef82e87, 51a574cb98a84e3aabf09db9a952b712; no voters:
I20240502 01:59:09.614459 6314 raft_consensus.cc:2781] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:09.614735 6314 raft_consensus.cc:686] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [term 2 LEADER]: Becoming Leader. State: Replica: 51a574cb98a84e3aabf09db9a952b712, State: Running, Role: LEADER
I20240502 01:59:09.614789 2496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd52517cbc5b43b798af83233b3759b9" candidate_uuid: "51a574cb98a84e3aabf09db9a952b712" candidate_term: 2 candidate_status { last_received { term: 1 index: 750 } } ignore_live_leader: true dest_uuid: "07c427880bd54215847186eab221fce5"
I20240502 01:59:09.614938 2496 raft_consensus.cc:3037] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:09.615114 6314 consensus_queue.cc:237] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 750, Committed index: 750, Last appended: 1.750, Last appended by leader: 750, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 750 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:09.616099 2496 raft_consensus.cc:2454] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a574cb98a84e3aabf09db9a952b712 in term 2.
I20240502 01:59:09.617579 982 catalog_manager.cc:5484] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 reported cstate change: term changed from 1 to 2, leader changed from 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) to 51a574cb98a84e3aabf09db9a952b712 (127.27.111.200). New cstate: current_term: 2 leader_uuid: "51a574cb98a84e3aabf09db9a952b712" committed_config { opid_index: 750 OBSOLETE_local: false peers { permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 } } peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:09.634944 2366 consensus_queue.cc:237] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 631, Committed index: 631, Last appended: 2.631, Last appended by leader: 482, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 632 OBSOLETE_local: false peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:09.636210 5536 raft_consensus.cc:1261] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712 [term 2 FOLLOWER]: Refusing update from remote peer 09401a31cfd046a78a67a2d51e290224: Log matching property violated. Preceding OpId in replica: term: 2 index: 631. Preceding OpId from leader: term: 2 index: 632. (index mismatch)
I20240502 01:59:09.636598 5633 consensus_queue.cc:1035] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 632, Last known committed idx: 631, Time since last communication: 0.000s
I20240502 01:59:09.638592 6338 raft_consensus.cc:2932] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 [term 2 LEADER]: Committing config change with OpId 2.632: config changed from index 631 to 632, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New config: { opid_index: 632 OBSOLETE_local: false peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:09.639015 5536 raft_consensus.cc:2932] T 609e71d9a50b4497a0410eb09734c169 P 51a574cb98a84e3aabf09db9a952b712 [term 2 FOLLOWER]: Committing config change with OpId 2.632: config changed from index 631 to 632, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New config: { opid_index: 632 OBSOLETE_local: false peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:09.641455 982 catalog_manager.cc:5484] T 609e71d9a50b4497a0410eb09734c169 P 09401a31cfd046a78a67a2d51e290224 reported cstate change: config changed from index 631 to 632, VOTER a571e02bbe774f45b80bb4383642c683 (127.27.111.193) evicted. New cstate: current_term: 2 leader_uuid: "09401a31cfd046a78a67a2d51e290224" committed_config { opid_index: 632 OBSOLETE_local: false peers { permanent_uuid: "09401a31cfd046a78a67a2d51e290224" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 43819 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:09.644114 1103 tablet_service.cc:1514] Processing DeleteTablet for tablet 609e71d9a50b4497a0410eb09734c169 with delete_type TABLET_DATA_TOMBSTONED (TS a571e02bbe774f45b80bb4383642c683 not found in new config with opid_index 632) from {username='slave'} at 127.0.0.1:51778
I20240502 01:59:09.644477 6339 tablet_replica.cc:330] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683: stopping tablet replica
I20240502 01:59:09.644642 6339 raft_consensus.cc:2229] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:09.644847 6339 raft_consensus.cc:2258] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:09.646296 6339 ts_tablet_manager.cc:1875] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:09.648404 6339 ts_tablet_manager.cc:1888] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.631
I20240502 01:59:09.648490 6339 log.cc:1196] T 609e71d9a50b4497a0410eb09734c169 P a571e02bbe774f45b80bb4383642c683: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/609e71d9a50b4497a0410eb09734c169
I20240502 01:59:09.649221 968 catalog_manager.cc:4830] TS a571e02bbe774f45b80bb4383642c683 (127.27.111.193:39473): tablet 609e71d9a50b4497a0410eb09734c169 (table rebalance_test_table_0 [id=a79d488e199a471f817d760819b107cd]) successfully deleted
I20240502 01:59:09.713083 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.715122 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:09.717126 1234 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:09.720736 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.738866 5516 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:09.742631 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:09.750485 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.764747 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.846678 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.846760 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.851830 1234 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:09.854416 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:09.860026 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:09.883607 5516 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:09.883783 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:09.884286 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.011961 2496 raft_consensus.cc:1261] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5 [term 2 FOLLOWER]: Refusing update from remote peer 51a574cb98a84e3aabf09db9a952b712: Log matching property violated. Preceding OpId in replica: term: 1 index: 750. Preceding OpId from leader: term: 2 index: 751. (index mismatch)
I20240502 01:59:10.012496 6315 consensus_queue.cc:1035] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 751, Last known committed idx: 750, Time since last communication: 0.000s
I20240502 01:59:10.016876 1254 raft_consensus.cc:1261] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Refusing update from remote peer 51a574cb98a84e3aabf09db9a952b712: Log matching property violated. Preceding OpId in replica: term: 1 index: 750. Preceding OpId from leader: term: 2 index: 751. (index mismatch)
I20240502 01:59:10.017411 6314 consensus_queue.cc:1035] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d8c8df1c10b41fab826db1b2ef82e87" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 751, Last known committed idx: 750, Time since last communication: 0.000s
I20240502 01:59:10.165503 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.168695 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.179782 1234 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:10.186568 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.192394 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:10.195231 5516 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:10.210853 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:10.213725 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.254684 5536 consensus_queue.cc:237] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 751, Committed index: 751, Last appended: 2.751, Last appended by leader: 750, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 752 OBSOLETE_local: false peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } }
I20240502 01:59:10.255789 2496 raft_consensus.cc:1261] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5 [term 2 FOLLOWER]: Refusing update from remote peer 51a574cb98a84e3aabf09db9a952b712: Log matching property violated. Preceding OpId in replica: term: 2 index: 751. Preceding OpId from leader: term: 2 index: 752. (index mismatch)
I20240502 01:59:10.256160 6314 consensus_queue.cc:1035] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 752, Last known committed idx: 751, Time since last communication: 0.000s
I20240502 01:59:10.257809 6315 raft_consensus.cc:2932] T dd52517cbc5b43b798af83233b3759b9 P 51a574cb98a84e3aabf09db9a952b712 [term 2 LEADER]: Committing config change with OpId 2.752: config changed from index 750 to 752, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New config: { opid_index: 752 OBSOLETE_local: false peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:10.258234 2496 raft_consensus.cc:2932] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5 [term 2 FOLLOWER]: Committing config change with OpId 2.752: config changed from index 750 to 752, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New config: { opid_index: 752 OBSOLETE_local: false peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:10.260555 975 catalog_manager.cc:5484] T dd52517cbc5b43b798af83233b3759b9 P 07c427880bd54215847186eab221fce5 reported cstate change: config changed from index 750 to 752, VOTER 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194) evicted. New cstate: current_term: 2 leader_uuid: "51a574cb98a84e3aabf09db9a952b712" committed_config { opid_index: 752 OBSOLETE_local: false peers { permanent_uuid: "07c427880bd54215847186eab221fce5" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 39309 } } peers { permanent_uuid: "51a574cb98a84e3aabf09db9a952b712" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 35643 } } }
I20240502 01:59:10.263366 1234 tablet_service.cc:1514] Processing DeleteTablet for tablet dd52517cbc5b43b798af83233b3759b9 with delete_type TABLET_DATA_TOMBSTONED (TS 4d8c8df1c10b41fab826db1b2ef82e87 not found in new config with opid_index 752) from {username='slave'} at 127.0.0.1:40794
I20240502 01:59:10.263675 6493 tablet_replica.cc:330] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87: stopping tablet replica
I20240502 01:59:10.263816 6493 raft_consensus.cc:2229] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:10.263919 6493 raft_consensus.cc:2258] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:10.265272 6493 ts_tablet_manager.cc:1875] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:10.267246 6493 ts_tablet_manager.cc:1888] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.751
I20240502 01:59:10.267315 6493 log.cc:1196] T dd52517cbc5b43b798af83233b3759b9 P 4d8c8df1c10b41fab826db1b2ef82e87: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/dd52517cbc5b43b798af83233b3759b9
I20240502 01:59:10.267944 967 catalog_manager.cc:4830] TS 4d8c8df1c10b41fab826db1b2ef82e87 (127.27.111.194:33267): tablet dd52517cbc5b43b798af83233b3759b9 (table rebalance_test_table_3 [id=f35b0e8f48d64f5884fb03539ed2621e]) successfully deleted
I20240502 01:59:10.526582 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:10.531652 5516 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:10.532915 1234 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:10.533772 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.535329 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.538591 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.545451 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:10.559618 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.661280 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:10.661798 1234 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:10.662339 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.662379 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:10.672253 5516 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:10.679766 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.683894 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.697046 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.794517 2479 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.794821 2347 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.800875 5516 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:10.803005 1234 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:10.814296 2608 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.814993 1103 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:10.815383 2743 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:10.839159 1366 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
8cfd9cbfd05e40b09dbec2dce9761c79 | 127.27.111.254:38251 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.27.111.212:36285 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
07c427880bd54215847186eab221fce5 | 127.27.111.197:39309 | HEALTHY | <none> | 2 | 0
09401a31cfd046a78a67a2d51e290224 | 127.27.111.196:43819 | HEALTHY | <none> | 2 | 0
37279883fafe432f81fe7b8224dbe5fb | 127.27.111.199:33361 | HEALTHY | <none> | 0 | 0
3f073f261d9447b99220aa532c2b1072 | 127.27.111.195:45227 | HEALTHY | <none> | 3 | 0
4d8c8df1c10b41fab826db1b2ef82e87 | 127.27.111.194:33267 | HEALTHY | <none> | 4 | 0
51a574cb98a84e3aabf09db9a952b712 | 127.27.111.200:35643 | HEALTHY | <none> | 1 | 0
72de8d04b745434f923594c920aff90c | 127.27.111.198:45287 | HEALTHY | <none> | 2 | 0
a571e02bbe774f45b80bb4383642c683 | 127.27.111.193:39473 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.27.111.193 | experimental | 127.27.111.193:39473
local_ip_for_outbound_sockets | 127.27.111.194 | experimental | 127.27.111.194:33267
local_ip_for_outbound_sockets | 127.27.111.195 | experimental | 127.27.111.195:45227
local_ip_for_outbound_sockets | 127.27.111.196 | experimental | 127.27.111.196:43819
local_ip_for_outbound_sockets | 127.27.111.197 | experimental | 127.27.111.197:39309
local_ip_for_outbound_sockets | 127.27.111.198 | experimental | 127.27.111.198:45287
local_ip_for_outbound_sockets | 127.27.111.199 | experimental | 127.27.111.199:33361
local_ip_for_outbound_sockets | 127.27.111.200 | experimental | 127.27.111.200:35643
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.27.111.193:39473
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.27.111.194:33267
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.27.111.195:45227
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.27.111.196:43819
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.27.111.197:39309
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.27.111.198:45287
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.27.111.199:33361
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.27.111.200:35643
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.27.111.212:36285 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 01:59:10.861267 28095 log_verifier.cc:126] Checking tablet 0fa2250275ba4eb391f88f9ee0fd52e9
I20240502 01:59:10.871179 28095 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 0fa2250275ba4eb391f88f9ee0fd52e9
I20240502 01:59:10.871253 28095 log_verifier.cc:126] Checking tablet 299019dc3d0848ad9324a84e319b9222
I20240502 01:59:10.971948 28095 log_verifier.cc:177] Verified matching terms for 637 ops in tablet 299019dc3d0848ad9324a84e319b9222
I20240502 01:59:10.972138 28095 log_verifier.cc:126] Checking tablet 5b90b275dd77400f9aaa1ffc0ea4be18
I20240502 01:59:11.061628 28095 log_verifier.cc:177] Verified matching terms for 575 ops in tablet 5b90b275dd77400f9aaa1ffc0ea4be18
I20240502 01:59:11.061806 28095 log_verifier.cc:126] Checking tablet 5bba25f4abe945429b0cc8ac93e12c8a
I20240502 01:59:11.202008 28095 log_verifier.cc:177] Verified matching terms for 693 ops in tablet 5bba25f4abe945429b0cc8ac93e12c8a
I20240502 01:59:11.202211 28095 log_verifier.cc:126] Checking tablet 609e71d9a50b4497a0410eb09734c169
I20240502 01:59:11.303679 28095 log_verifier.cc:177] Verified matching terms for 632 ops in tablet 609e71d9a50b4497a0410eb09734c169
I20240502 01:59:11.303903 28095 log_verifier.cc:126] Checking tablet 690f9ec1b4c34a919a9a09ae7c618ef3
I20240502 01:59:11.397370 28095 log_verifier.cc:177] Verified matching terms for 553 ops in tablet 690f9ec1b4c34a919a9a09ae7c618ef3
I20240502 01:59:11.397578 28095 log_verifier.cc:126] Checking tablet 82337220b48d45fba654828a31eabbf4
I20240502 01:59:11.495380 28095 log_verifier.cc:177] Verified matching terms for 634 ops in tablet 82337220b48d45fba654828a31eabbf4
I20240502 01:59:11.495584 28095 log_verifier.cc:126] Checking tablet 844edd610a734b0994f59a4b0e29af0b
I20240502 01:59:11.599459 28095 log_verifier.cc:177] Verified matching terms for 666 ops in tablet 844edd610a734b0994f59a4b0e29af0b
I20240502 01:59:11.599654 28095 log_verifier.cc:126] Checking tablet 90948f77e44241e1883249fbcdcf1515
I20240502 01:59:11.716671 28095 log_verifier.cc:177] Verified matching terms for 757 ops in tablet 90948f77e44241e1883249fbcdcf1515
I20240502 01:59:11.716908 28095 log_verifier.cc:126] Checking tablet a59a5084412f4082bdd35e7c546e71b4
I20240502 01:59:11.804913 28095 log_verifier.cc:177] Verified matching terms for 557 ops in tablet a59a5084412f4082bdd35e7c546e71b4
I20240502 01:59:11.805088 28095 log_verifier.cc:126] Checking tablet a6da3730570e48d2aba7457e9d41dfcc
I20240502 01:59:11.913038 28095 log_verifier.cc:177] Verified matching terms for 726 ops in tablet a6da3730570e48d2aba7457e9d41dfcc
I20240502 01:59:11.913290 28095 log_verifier.cc:126] Checking tablet c0e16afa25404fa78e03547f7b140283
I20240502 01:59:12.019814 28095 log_verifier.cc:177] Verified matching terms for 664 ops in tablet c0e16afa25404fa78e03547f7b140283
I20240502 01:59:12.020048 28095 log_verifier.cc:126] Checking tablet c8a946d0b927414f88c615d54ce5fc7f
I20240502 01:59:12.117355 28095 log_verifier.cc:177] Verified matching terms for 627 ops in tablet c8a946d0b927414f88c615d54ce5fc7f
I20240502 01:59:12.117575 28095 log_verifier.cc:126] Checking tablet d0cde29604ac4292be1ead0a69aa6908
I20240502 01:59:12.219767 28095 log_verifier.cc:177] Verified matching terms for 635 ops in tablet d0cde29604ac4292be1ead0a69aa6908
I20240502 01:59:12.219985 28095 log_verifier.cc:126] Checking tablet dd52517cbc5b43b798af83233b3759b9
I20240502 01:59:12.335680 28095 log_verifier.cc:177] Verified matching terms for 752 ops in tablet dd52517cbc5b43b798af83233b3759b9
I20240502 01:59:12.335924 28095 log_verifier.cc:126] Checking tablet fa93c869b01b4af69678e378df7db684
I20240502 01:59:12.438369 28095 log_verifier.cc:177] Verified matching terms for 654 ops in tablet fa93c869b01b4af69678e378df7db684
I20240502 01:59:12.440126 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 1043
I20240502 01:59:12.449738 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 1174
I20240502 01:59:12.457593 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 1305
I20240502 01:59:12.465440 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 2285
I20240502 01:59:12.472456 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 2416
I20240502 01:59:12.479126 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 2549
I20240502 01:59:12.485675 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 2681
I20240502 01:59:12.492214 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 5454
I20240502 01:59:12.498577 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 952
2024-05-02T01:59:12Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (39411 ms)
[----------] 1 test from RebalanceParamTest (39411 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240502 01:59:12.558508 28095 test_util.cc:268] Using random seed: -216514139
I20240502 01:59:12.559321 28095 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:59:12.559365 28095 ts_itest-base.cc:113] --------------
I20240502 01:59:12.559386 28095 ts_itest-base.cc:114] 8 tablet servers
I20240502 01:59:12.559402 28095 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:59:12.559419 28095 ts_itest-base.cc:116] --------------
2024-05-02T01:59:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:59:12Z Disabled control of system clock
I20240502 01:59:12.568333 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.111.254:44795
--webserver_interface=127.27.111.254
--webserver_port=0
--builtin_ntp_servers=127.27.111.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.111.254:44795
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 01:59:12.681973 6612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:12.682243 6612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:59:12.682286 6612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:12.685561 6612 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:59:12.685657 6612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:59:12.685681 6612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:59:12.685699 6612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:59:12.685717 6612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:59:12.689589 6612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.111.254:44795
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.111.254:44795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.111.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:59:12.690618 6612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:59:12.691787 6612 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:12.700085 6618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:12.700564 6617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:12.701455 6620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:12.702304 6612 server_base.cc:1034] running on GCE node
I20240502 01:59:12.702935 6612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:59:12.703992 6612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:59:12.705175 6612 hybrid_clock.cc:648] HybridClock initialized: now 1714615152705161 us; error 33 us; skew 500 ppm
I20240502 01:59:12.707401 6612 webserver.cc:460] Webserver started at http://127.27.111.254:40465/ using document root <none> and password file <none>
I20240502 01:59:12.708021 6612 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:12.708088 6612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:12.708302 6612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:12.710096 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8f0a4860df0c4e4a97901c87e80d6420"
format_stamp: "Formatted at 2024-05-02 01:59:12 on dist-test-slave-0pw1"
I20240502 01:59:12.710639 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8f0a4860df0c4e4a97901c87e80d6420"
format_stamp: "Formatted at 2024-05-02 01:59:12 on dist-test-slave-0pw1"
I20240502 01:59:12.714330 6612 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:59:12.716797 6626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:12.717949 6612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:12.718084 6612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8f0a4860df0c4e4a97901c87e80d6420"
format_stamp: "Formatted at 2024-05-02 01:59:12 on dist-test-slave-0pw1"
I20240502 01:59:12.718184 6612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:12.728328 6612 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:12.729001 6612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:59:12.729163 6612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:12.735926 6678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.254:44795 every 8 connection(s)
I20240502 01:59:12.735929 6612 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.254:44795
I20240502 01:59:12.737030 6612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:59:12.739782 6679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:12.744339 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 6612
I20240502 01:59:12.744491 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:59:12.745344 6679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420: Bootstrap starting.
I20240502 01:59:12.747789 6679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:12.748692 6679 log.cc:824] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420: Log is configured to *not* fsync() on all Append() calls
I20240502 01:59:12.750622 6679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420: No bootstrap required, opened a new log
I20240502 01:59:12.753461 6679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0a4860df0c4e4a97901c87e80d6420" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 44795 } }
I20240502 01:59:12.753661 6679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:12.753728 6679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0a4860df0c4e4a97901c87e80d6420, State: Initialized, Role: FOLLOWER
I20240502 01:59:12.754212 6679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0a4860df0c4e4a97901c87e80d6420" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 44795 } }
I20240502 01:59:12.754350 6679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:59:12.754439 6679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:59:12.754551 6679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:12.755360 6679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0a4860df0c4e4a97901c87e80d6420" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 44795 } }
I20240502 01:59:12.755687 6679 leader_election.cc:302] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f0a4860df0c4e4a97901c87e80d6420; no voters:
I20240502 01:59:12.755972 6679 leader_election.cc:288] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:59:12.756654 6684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:12.756861 6684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [term 1 LEADER]: Becoming Leader. State: Replica: 8f0a4860df0c4e4a97901c87e80d6420, State: Running, Role: LEADER
I20240502 01:59:12.757145 6679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:59:12.757226 6684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0a4860df0c4e4a97901c87e80d6420" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 44795 } }
I20240502 01:59:12.758775 6686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0a4860df0c4e4a97901c87e80d6420. Latest consensus state: current_term: 1 leader_uuid: "8f0a4860df0c4e4a97901c87e80d6420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0a4860df0c4e4a97901c87e80d6420" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 44795 } } }
I20240502 01:59:12.758846 6685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f0a4860df0c4e4a97901c87e80d6420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0a4860df0c4e4a97901c87e80d6420" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 44795 } } }
I20240502 01:59:12.758936 6686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [sys.catalog]: This master's current role is: LEADER
I20240502 01:59:12.758947 6685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420 [sys.catalog]: This master's current role is: LEADER
I20240502 01:59:12.759318 6693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:59:12.762015 6693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:59:12.767166 6693 catalog_manager.cc:1347] Generated new cluster ID: 8da6371052d742369cb5fc0114b44c9d
I20240502 01:59:12.767264 6693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:59:12.801796 6693 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:59:12.802896 6693 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:59:12.808602 6693 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8f0a4860df0c4e4a97901c87e80d6420: Generated new TSK 0
I20240502 01:59:12.809357 6693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:59:12.817668 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.193:0
--local_ip_for_outbound_sockets=127.27.111.193
--webserver_interface=127.27.111.193
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--builtin_ntp_servers=127.27.111.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:59:12.934022 6703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:12.934280 6703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:59:12.934330 6703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:12.937522 6703 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:59:12.937611 6703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:59:12.937685 6703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.193
I20240502 01:59:12.941524 6703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.111.193
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--never_fsync=true
--heap_profile_path=/tmp/kudu.6703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.193
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:59:12.942561 6703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:59:12.943796 6703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:12.952718 6708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:12.953280 6709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:12.954078 6711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:12.954926 6703 server_base.cc:1034] running on GCE node
I20240502 01:59:12.955335 6703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:59:12.955945 6703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:59:12.957096 6703 hybrid_clock.cc:648] HybridClock initialized: now 1714615152957081 us; error 31 us; skew 500 ppm
I20240502 01:59:12.959419 6703 webserver.cc:460] Webserver started at http://127.27.111.193:35069/ using document root <none> and password file <none>
I20240502 01:59:12.960047 6703 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:12.960110 6703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:12.960310 6703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:12.962095 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "85e78945910a4e1abfa101d2edc3644e"
format_stamp: "Formatted at 2024-05-02 01:59:12 on dist-test-slave-0pw1"
I20240502 01:59:12.962636 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "85e78945910a4e1abfa101d2edc3644e"
format_stamp: "Formatted at 2024-05-02 01:59:12 on dist-test-slave-0pw1"
I20240502 01:59:12.966176 6703 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:59:12.968621 6717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:12.969815 6703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:12.969959 6703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "85e78945910a4e1abfa101d2edc3644e"
format_stamp: "Formatted at 2024-05-02 01:59:12 on dist-test-slave-0pw1"
I20240502 01:59:12.970062 6703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:13.011123 6703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:13.011818 6703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:59:13.012003 6703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:13.012562 6703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:13.013532 6703 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:13.013583 6703 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.013645 6703 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:13.013671 6703 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.022624 6703 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.193:42261
I20240502 01:59:13.022660 6830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.193:42261 every 8 connection(s)
I20240502 01:59:13.023664 6703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:59:13.024780 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 6703
I20240502 01:59:13.024940 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:59:13.027073 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.194:0
--local_ip_for_outbound_sockets=127.27.111.194
--webserver_interface=127.27.111.194
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--builtin_ntp_servers=127.27.111.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:59:13.033356 6831 heartbeater.cc:347] Connected to a master server at 127.27.111.254:44795
I20240502 01:59:13.033592 6831 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:13.034242 6831 heartbeater.cc:511] Master 127.27.111.254:44795 requested a full tablet report, sending...
I20240502 01:59:13.035828 6643 ts_manager.cc:194] Registered new tserver with Master: 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:13.037067 6643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.193:43933
W20240502 01:59:13.141453 6834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:13.141750 6834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:59:13.141835 6834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:13.145088 6834 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:59:13.145192 6834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:59:13.145290 6834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.194
I20240502 01:59:13.149444 6834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.111.194
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--never_fsync=true
--heap_profile_path=/tmp/kudu.6834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.194
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:59:13.150496 6834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:59:13.151645 6834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:13.160727 6840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:13.161568 6839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:13.162290 6842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:13.162986 6834 server_base.cc:1034] running on GCE node
I20240502 01:59:13.163473 6834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:59:13.164095 6834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:59:13.165266 6834 hybrid_clock.cc:648] HybridClock initialized: now 1714615153165258 us; error 30 us; skew 500 ppm
I20240502 01:59:13.167505 6834 webserver.cc:460] Webserver started at http://127.27.111.194:46815/ using document root <none> and password file <none>
I20240502 01:59:13.168112 6834 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:13.168175 6834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:13.168373 6834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:13.170174 6834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.170709 6834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.174379 6834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:59:13.176975 6848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.178148 6834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:13.178284 6834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.178445 6834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:13.200302 6834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:13.201006 6834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:59:13.201175 6834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:13.201802 6834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:13.202795 6834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:13.202848 6834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.202894 6834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:13.202925 6834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.211467 6834 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.194:44305
I20240502 01:59:13.211498 6962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.194:44305 every 8 connection(s)
I20240502 01:59:13.212442 6834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:59:13.213387 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 6834
I20240502 01:59:13.213496 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:59:13.215685 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.195:0
--local_ip_for_outbound_sockets=127.27.111.195
--webserver_interface=127.27.111.195
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--builtin_ntp_servers=127.27.111.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:59:13.221999 6963 heartbeater.cc:347] Connected to a master server at 127.27.111.254:44795
I20240502 01:59:13.222254 6963 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:13.222774 6963 heartbeater.cc:511] Master 127.27.111.254:44795 requested a full tablet report, sending...
I20240502 01:59:13.223759 6643 ts_manager.cc:194] Registered new tserver with Master: e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:13.224296 6643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.194:45797
W20240502 01:59:13.328539 6966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:13.328824 6966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:59:13.328900 6966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:13.332149 6966 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:59:13.332233 6966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:59:13.332330 6966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.195
I20240502 01:59:13.336089 6966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.111.195
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--never_fsync=true
--heap_profile_path=/tmp/kudu.6966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.195
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:59:13.337275 6966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:59:13.338632 6966 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:13.347723 6974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:13.348220 6972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:13.349017 6971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:13.349851 6966 server_base.cc:1034] running on GCE node
I20240502 01:59:13.350317 6966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:59:13.350993 6966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:59:13.352162 6966 hybrid_clock.cc:648] HybridClock initialized: now 1714615153352138 us; error 47 us; skew 500 ppm
I20240502 01:59:13.354271 6966 webserver.cc:460] Webserver started at http://127.27.111.195:42405/ using document root <none> and password file <none>
I20240502 01:59:13.354882 6966 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:13.354948 6966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:13.355154 6966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:13.356889 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c6101231591a458384f4ea6d1f0e0ae5"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.357513 6966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c6101231591a458384f4ea6d1f0e0ae5"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.361631 6966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 01:59:13.364109 6981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.365166 6966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:13.365279 6966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c6101231591a458384f4ea6d1f0e0ae5"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.365370 6966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:13.396571 6966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:13.397253 6966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:59:13.397430 6966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:13.398012 6966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:13.399006 6966 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:13.399057 6966 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.399119 6966 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:13.399144 6966 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.407862 6966 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.195:39125
I20240502 01:59:13.407899 7095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.195:39125 every 8 connection(s)
I20240502 01:59:13.408913 6966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:59:13.412343 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 6966
I20240502 01:59:13.412473 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:59:13.414798 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.196:0
--local_ip_for_outbound_sockets=127.27.111.196
--webserver_interface=127.27.111.196
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--builtin_ntp_servers=127.27.111.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:59:13.418365 7096 heartbeater.cc:347] Connected to a master server at 127.27.111.254:44795
I20240502 01:59:13.418623 7096 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:13.419109 7096 heartbeater.cc:511] Master 127.27.111.254:44795 requested a full tablet report, sending...
I20240502 01:59:13.420173 6643 ts_manager.cc:194] Registered new tserver with Master: c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:13.421037 6643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.195:47905
W20240502 01:59:13.529706 7099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:13.529994 7099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:59:13.530040 7099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:13.533427 7099 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:59:13.533529 7099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:59:13.533663 7099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.196
I20240502 01:59:13.537710 7099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.111.196
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--never_fsync=true
--heap_profile_path=/tmp/kudu.7099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.196
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:59:13.538790 7099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:59:13.540040 7099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:13.549023 7107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:13.549504 7105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:13.550292 7104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:13.551146 7099 server_base.cc:1034] running on GCE node
I20240502 01:59:13.551585 7099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:59:13.552242 7099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:59:13.553414 7099 hybrid_clock.cc:648] HybridClock initialized: now 1714615153553393 us; error 36 us; skew 500 ppm
I20240502 01:59:13.555598 7099 webserver.cc:460] Webserver started at http://127.27.111.196:43905/ using document root <none> and password file <none>
I20240502 01:59:13.556207 7099 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:13.556267 7099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:13.556454 7099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:13.558310 7099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "86bc8001c3464b929cd5c873861baa0c"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.558893 7099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "86bc8001c3464b929cd5c873861baa0c"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.562588 7099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:59:13.565057 7113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.566089 7099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:13.566211 7099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "86bc8001c3464b929cd5c873861baa0c"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.566303 7099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:13.580041 7099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:13.580699 7099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:59:13.580873 7099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:13.581485 7099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:13.582468 7099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:13.582516 7099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.582556 7099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:13.582577 7099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.593827 7099 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.196:36569
I20240502 01:59:13.593874 7227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.196:36569 every 8 connection(s)
I20240502 01:59:13.594816 7099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:59:13.601656 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 7099
I20240502 01:59:13.601845 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:59:13.603827 7228 heartbeater.cc:347] Connected to a master server at 127.27.111.254:44795
I20240502 01:59:13.604115 7228 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:13.604256 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.197:0
--local_ip_for_outbound_sockets=127.27.111.197
--webserver_interface=127.27.111.197
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--builtin_ntp_servers=127.27.111.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:59:13.604714 7228 heartbeater.cc:511] Master 127.27.111.254:44795 requested a full tablet report, sending...
I20240502 01:59:13.605926 6642 ts_manager.cc:194] Registered new tserver with Master: 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:13.606707 6642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.196:46083
W20240502 01:59:13.717373 7231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:13.717617 7231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:59:13.717664 7231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:13.720888 7231 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:59:13.720973 7231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:59:13.721057 7231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.197
I20240502 01:59:13.724653 7231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.111.197
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--never_fsync=true
--heap_profile_path=/tmp/kudu.7231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.197
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:59:13.725586 7231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:59:13.726794 7231 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:13.735829 7236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:13.736482 7239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:13.737187 7237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:13.737937 7231 server_base.cc:1034] running on GCE node
I20240502 01:59:13.738444 7231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:59:13.739199 7231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:59:13.740343 7231 hybrid_clock.cc:648] HybridClock initialized: now 1714615153740318 us; error 40 us; skew 500 ppm
I20240502 01:59:13.742461 7231 webserver.cc:460] Webserver started at http://127.27.111.197:38865/ using document root <none> and password file <none>
I20240502 01:59:13.743083 7231 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:13.743149 7231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:13.743341 7231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:13.745044 7231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "954cd49326bf405abd81182d75dc783c"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.745561 7231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "954cd49326bf405abd81182d75dc783c"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.749022 7231 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 01:59:13.751308 7245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.752389 7231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:59:13.752499 7231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "954cd49326bf405abd81182d75dc783c"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.752585 7231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:13.773774 7231 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:13.774542 7231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:59:13.774716 7231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:13.775290 7231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:13.776224 7231 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:13.776274 7231 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.776333 7231 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:13.776364 7231 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.784968 7231 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.197:45767
I20240502 01:59:13.784996 7360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.197:45767 every 8 connection(s)
I20240502 01:59:13.785987 7231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:59:13.791358 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 7231
I20240502 01:59:13.791496 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:59:13.794121 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.198:0
--local_ip_for_outbound_sockets=127.27.111.198
--webserver_interface=127.27.111.198
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--builtin_ntp_servers=127.27.111.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:59:13.795392 7361 heartbeater.cc:347] Connected to a master server at 127.27.111.254:44795
I20240502 01:59:13.795629 7361 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:13.796097 7361 heartbeater.cc:511] Master 127.27.111.254:44795 requested a full tablet report, sending...
I20240502 01:59:13.797112 6642 ts_manager.cc:194] Registered new tserver with Master: 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767)
I20240502 01:59:13.797961 6642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.197:37123
W20240502 01:59:13.904913 7364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:13.905172 7364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:59:13.905222 7364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:13.908355 7364 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:59:13.908435 7364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:59:13.908535 7364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.198
I20240502 01:59:13.912402 7364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.111.198
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--never_fsync=true
--heap_profile_path=/tmp/kudu.7364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.198
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:59:13.913429 7364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:59:13.914638 7364 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:13.923413 7369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:13.923887 7372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:13.924427 7370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:13.925434 7364 server_base.cc:1034] running on GCE node
I20240502 01:59:13.925863 7364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:59:13.926506 7364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:59:13.927676 7364 hybrid_clock.cc:648] HybridClock initialized: now 1714615153927661 us; error 29 us; skew 500 ppm
I20240502 01:59:13.929787 7364 webserver.cc:460] Webserver started at http://127.27.111.198:40441/ using document root <none> and password file <none>
I20240502 01:59:13.930383 7364 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:13.930482 7364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:13.930676 7364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:13.932478 7364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "4f53d80e39be410d92d4766692aff1c6"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.932956 7364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "4f53d80e39be410d92d4766692aff1c6"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.936458 7364 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:59:13.938795 7378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.939846 7364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:59:13.939989 7364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4f53d80e39be410d92d4766692aff1c6"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:13.940086 7364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:13.953841 7364 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:13.954519 7364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:59:13.954680 7364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:13.955221 7364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:13.956310 7364 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:13.956364 7364 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.956427 7364 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:13.956455 7364 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:13.965481 7364 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.198:46475
I20240502 01:59:13.965508 7491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.198:46475 every 8 connection(s)
I20240502 01:59:13.966475 7364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:59:13.970942 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 7364
I20240502 01:59:13.971060 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:59:13.973179 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.199:0
--local_ip_for_outbound_sockets=127.27.111.199
--webserver_interface=127.27.111.199
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--builtin_ntp_servers=127.27.111.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:59:13.975589 7492 heartbeater.cc:347] Connected to a master server at 127.27.111.254:44795
I20240502 01:59:13.975823 7492 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:13.976416 7492 heartbeater.cc:511] Master 127.27.111.254:44795 requested a full tablet report, sending...
I20240502 01:59:13.977519 6642 ts_manager.cc:194] Registered new tserver with Master: 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
I20240502 01:59:13.978361 6642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.198:57153
I20240502 01:59:14.039532 6831 heartbeater.cc:503] Master 127.27.111.254:44795 was elected leader, sending a full tablet report...
W20240502 01:59:14.088979 7495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:14.089223 7495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:59:14.089272 7495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:14.092432 7495 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:59:14.092509 7495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:59:14.092581 7495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.199
I20240502 01:59:14.096266 7495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.111.199
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--never_fsync=true
--heap_profile_path=/tmp/kudu.7495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.199
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:59:14.097198 7495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:59:14.098313 7495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:14.107138 7500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:14.107689 7503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:14.108589 7501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:14.109288 7495 server_base.cc:1034] running on GCE node
I20240502 01:59:14.109737 7495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:59:14.110368 7495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:59:14.111552 7495 hybrid_clock.cc:648] HybridClock initialized: now 1714615154111537 us; error 32 us; skew 500 ppm
I20240502 01:59:14.113639 7495 webserver.cc:460] Webserver started at http://127.27.111.199:45973/ using document root <none> and password file <none>
I20240502 01:59:14.114281 7495 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:14.114343 7495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:14.114563 7495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:14.116295 7495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2"
format_stamp: "Formatted at 2024-05-02 01:59:14 on dist-test-slave-0pw1"
I20240502 01:59:14.116762 7495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2"
format_stamp: "Formatted at 2024-05-02 01:59:14 on dist-test-slave-0pw1"
I20240502 01:59:14.120389 7495 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:59:14.122699 7509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:14.123703 7495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 01:59:14.123801 7495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2"
format_stamp: "Formatted at 2024-05-02 01:59:14 on dist-test-slave-0pw1"
I20240502 01:59:14.123884 7495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:14.139843 7495 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:14.140508 7495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:59:14.140661 7495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:14.141177 7495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:14.142072 7495 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:14.142125 7495 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:14.142175 7495 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:14.142197 7495 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:14.150910 7495 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.199:41741
I20240502 01:59:14.151021 7623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.199:41741 every 8 connection(s)
I20240502 01:59:14.151937 7495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:59:14.160549 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 7495
I20240502 01:59:14.160696 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:59:14.161470 7624 heartbeater.cc:347] Connected to a master server at 127.27.111.254:44795
I20240502 01:59:14.161718 7624 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:14.162251 7624 heartbeater.cc:511] Master 127.27.111.254:44795 requested a full tablet report, sending...
I20240502 01:59:14.162917 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.200:0
--local_ip_for_outbound_sockets=127.27.111.200
--webserver_interface=127.27.111.200
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--builtin_ntp_servers=127.27.111.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:59:14.163326 6642 ts_manager.cc:194] Registered new tserver with Master: 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741)
I20240502 01:59:14.163877 6642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.199:54383
I20240502 01:59:14.226238 6963 heartbeater.cc:503] Master 127.27.111.254:44795 was elected leader, sending a full tablet report...
W20240502 01:59:14.274085 7627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:14.274333 7627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:59:14.274435 7627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:14.277482 7627 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:59:14.277562 7627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:59:14.277658 7627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.200
I20240502 01:59:14.281315 7627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.111.200
--webserver_port=0
--tserver_master_addrs=127.27.111.254:44795
--never_fsync=true
--heap_profile_path=/tmp/kudu.7627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.200
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:59:14.282289 7627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:59:14.283600 7627 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:14.292317 7633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:14.293007 7634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:14.293737 7636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:14.294442 7627 server_base.cc:1034] running on GCE node
I20240502 01:59:14.294855 7627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:59:14.295462 7627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:59:14.296630 7627 hybrid_clock.cc:648] HybridClock initialized: now 1714615154296616 us; error 29 us; skew 500 ppm
I20240502 01:59:14.298667 7627 webserver.cc:460] Webserver started at http://127.27.111.200:41449/ using document root <none> and password file <none>
I20240502 01:59:14.299324 7627 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:14.299384 7627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:14.299585 7627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:59:14.301275 7627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "070558c278e0457b9cc070ac0d662b5d"
format_stamp: "Formatted at 2024-05-02 01:59:14 on dist-test-slave-0pw1"
I20240502 01:59:14.301756 7627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "070558c278e0457b9cc070ac0d662b5d"
format_stamp: "Formatted at 2024-05-02 01:59:14 on dist-test-slave-0pw1"
I20240502 01:59:14.305070 7627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 01:59:14.307374 7642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:14.308459 7627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:59:14.308570 7627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "070558c278e0457b9cc070ac0d662b5d"
format_stamp: "Formatted at 2024-05-02 01:59:14 on dist-test-slave-0pw1"
I20240502 01:59:14.308663 7627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:14.332068 7627 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:14.332705 7627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:59:14.332866 7627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:14.333475 7627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:14.334409 7627 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:14.334467 7627 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:14.334509 7627 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:14.334530 7627 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:14.342685 7627 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.200:37235
I20240502 01:59:14.342712 7756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.200:37235 every 8 connection(s)
I20240502 01:59:14.343690 7627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:59:14.349395 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 7627
I20240502 01:59:14.349558 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:59:14.352623 7757 heartbeater.cc:347] Connected to a master server at 127.27.111.254:44795
I20240502 01:59:14.352842 7757 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:14.353307 7757 heartbeater.cc:511] Master 127.27.111.254:44795 requested a full tablet report, sending...
I20240502 01:59:14.354173 6642 ts_manager.cc:194] Registered new tserver with Master: 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
I20240502 01:59:14.354736 6642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.200:50115
I20240502 01:59:14.363940 28095 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:59:14.373369 28095 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:59:14.373450 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 7231
I20240502 01:59:14.379071 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 7364
I20240502 01:59:14.384905 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 7495
I20240502 01:59:14.390327 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 7627
I20240502 01:59:14.423336 7096 heartbeater.cc:503] Master 127.27.111.254:44795 was elected leader, sending a full tablet report...
I20240502 01:59:14.609553 7228 heartbeater.cc:503] Master 127.27.111.254:44795 was elected leader, sending a full tablet report...
I20240502 01:59:18.147493 6642 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32904:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:18.168715 7029 tablet_service.cc:1467] Processing CreateTablet for tablet b69d6e907b3b400f93721981732c006b (DEFAULT_TABLE table=rebalance_test_table_0 [id=6397b955acf741aeb8f869ab987d74b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.168715 6896 tablet_service.cc:1467] Processing CreateTablet for tablet b69d6e907b3b400f93721981732c006b (DEFAULT_TABLE table=rebalance_test_table_0 [id=6397b955acf741aeb8f869ab987d74b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.168720 6764 tablet_service.cc:1467] Processing CreateTablet for tablet a66b932d884e4df8a78841103d88ded0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6397b955acf741aeb8f869ab987d74b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.169699 7029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69d6e907b3b400f93721981732c006b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.169700 6764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a66b932d884e4df8a78841103d88ded0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.169710 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69d6e907b3b400f93721981732c006b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.170656 6894 tablet_service.cc:1467] Processing CreateTablet for tablet a66b932d884e4df8a78841103d88ded0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6397b955acf741aeb8f869ab987d74b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.170490 7161 tablet_service.cc:1467] Processing CreateTablet for tablet dcc5daef41bc44c4855f2bcb44e1c6d5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6397b955acf741aeb8f869ab987d74b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.170889 6894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a66b932d884e4df8a78841103d88ded0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.171272 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc5daef41bc44c4855f2bcb44e1c6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.173491 6897 tablet_service.cc:1467] Processing CreateTablet for tablet dcc5daef41bc44c4855f2bcb44e1c6d5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6397b955acf741aeb8f869ab987d74b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.173743 6897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc5daef41bc44c4855f2bcb44e1c6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.174686 6765 tablet_service.cc:1467] Processing CreateTablet for tablet b69d6e907b3b400f93721981732c006b (DEFAULT_TABLE table=rebalance_test_table_0 [id=6397b955acf741aeb8f869ab987d74b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.174950 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69d6e907b3b400f93721981732c006b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.174897 7160 tablet_service.cc:1467] Processing CreateTablet for tablet a66b932d884e4df8a78841103d88ded0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6397b955acf741aeb8f869ab987d74b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.175135 7160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a66b932d884e4df8a78841103d88ded0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.168720 7030 tablet_service.cc:1467] Processing CreateTablet for tablet dcc5daef41bc44c4855f2bcb44e1c6d5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6397b955acf741aeb8f869ab987d74b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.177174 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc5daef41bc44c4855f2bcb44e1c6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.178745 7779 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:18.180464 7779 tablet_bootstrap.cc:654] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.181279 7779 log.cc:824] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c: Log is configured to *not* fsync() on all Append() calls
I20240502 01:59:18.182077 7778 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:18.183046 7779 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:18.183132 7777 tablet_bootstrap.cc:492] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:18.183256 7779 ts_tablet_manager.cc:1367] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:59:18.184135 7778 tablet_bootstrap.cc:654] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.184885 7778 log.cc:824] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8: Log is configured to *not* fsync() on all Append() calls
I20240502 01:59:18.184998 7777 tablet_bootstrap.cc:654] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.185535 7780 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:18.185707 7777 log.cc:824] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e: Log is configured to *not* fsync() on all Append() calls
I20240502 01:59:18.186709 7779 raft_consensus.cc:348] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.186946 7779 raft_consensus.cc:374] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.187005 7779 raft_consensus.cc:729] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.187415 7777 tablet_bootstrap.cc:492] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:18.187424 7778 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:18.187433 7780 tablet_bootstrap.cc:654] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.187563 7777 ts_tablet_manager.cc:1367] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:59:18.187484 7779 consensus_queue.cc:260] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.187578 7778 ts_tablet_manager.cc:1367] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:59:18.188138 7780 log.cc:824] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5: Log is configured to *not* fsync() on all Append() calls
I20240502 01:59:18.188328 7779 ts_tablet_manager.cc:1398] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 01:59:18.188671 7779 tablet_bootstrap.cc:492] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:18.189690 7779 tablet_bootstrap.cc:654] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.190886 7779 tablet_bootstrap.cc:492] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:18.190976 7779 ts_tablet_manager.cc:1367] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:18.191171 7778 raft_consensus.cc:348] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.191380 7778 raft_consensus.cc:374] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.191445 7778 raft_consensus.cc:729] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.191711 7779 raft_consensus.cc:348] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:18.191821 7779 raft_consensus.cc:374] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.191866 7779 raft_consensus.cc:729] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.191902 7778 consensus_queue.cc:260] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.192030 7779 consensus_queue.cc:260] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:18.192426 7779 ts_tablet_manager.cc:1398] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:18.192713 7778 ts_tablet_manager.cc:1398] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240502 01:59:18.192903 7780 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:18.192878 7777 raft_consensus.cc:348] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:18.193044 7777 raft_consensus.cc:374] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.193048 7778 tablet_bootstrap.cc:492] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:18.193059 7780 ts_tablet_manager.cc:1367] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 01:59:18.193100 7777 raft_consensus.cc:729] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.193517 7777 consensus_queue.cc:260] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:18.193957 7778 tablet_bootstrap.cc:654] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.194332 7777 ts_tablet_manager.cc:1398] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:59:18.194712 7777 tablet_bootstrap.cc:492] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:18.195956 7777 tablet_bootstrap.cc:654] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.196830 7780 raft_consensus.cc:348] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.197319 7780 raft_consensus.cc:374] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.197396 7780 raft_consensus.cc:729] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.197592 7778 tablet_bootstrap.cc:492] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:18.197666 7778 ts_tablet_manager.cc:1367] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:59:18.197795 7780 consensus_queue.cc:260] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.198210 7778 raft_consensus.cc:348] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.198313 7778 raft_consensus.cc:374] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.198343 7778 raft_consensus.cc:729] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.198503 7778 consensus_queue.cc:260] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.198758 7778 ts_tablet_manager.cc:1398] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:18.198784 7777 tablet_bootstrap.cc:492] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:18.198794 7780 ts_tablet_manager.cc:1398] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:59:18.198925 7778 tablet_bootstrap.cc:492] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:18.199713 7778 tablet_bootstrap.cc:654] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.199775 7777 ts_tablet_manager.cc:1367] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:18.199898 7780 tablet_bootstrap.cc:492] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:18.200373 7777 raft_consensus.cc:348] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.200471 7777 raft_consensus.cc:374] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.200503 7777 raft_consensus.cc:729] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.200621 7777 consensus_queue.cc:260] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.200910 7780 tablet_bootstrap.cc:654] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.200932 7777 ts_tablet_manager.cc:1398] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:18.201189 7778 tablet_bootstrap.cc:492] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:18.201261 7778 ts_tablet_manager.cc:1367] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:18.201730 7778 raft_consensus.cc:348] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:18.201817 7778 raft_consensus.cc:374] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.201865 7778 raft_consensus.cc:729] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.201967 7780 tablet_bootstrap.cc:492] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:18.201977 7778 consensus_queue.cc:260] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:18.202033 7780 ts_tablet_manager.cc:1367] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:18.202221 7778 ts_tablet_manager.cc:1398] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:18.202594 7780 raft_consensus.cc:348] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.202710 7780 raft_consensus.cc:374] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.202752 7780 raft_consensus.cc:729] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.202881 7780 consensus_queue.cc:260] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.203146 7780 ts_tablet_manager.cc:1398] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 01:59:18.217206 6964 tablet.cc:2369] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:59:18.236775 7789 raft_consensus.cc:482] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:18.236995 7789 raft_consensus.cc:504] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:18.238121 7789 leader_election.cc:288] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:18.241863 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66b932d884e4df8a78841103d88ded0" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:18.241885 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66b932d884e4df8a78841103d88ded0" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:18.242172 7182 raft_consensus.cc:2454] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:18.242172 6917 raft_consensus.cc:2454] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:18.242663 6721 leader_election.cc:302] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:18.242913 7789 raft_consensus.cc:2781] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:18.242996 7789 raft_consensus.cc:482] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:18.243026 7789 raft_consensus.cc:3037] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.243885 7789 raft_consensus.cc:504] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:18.244197 7789 leader_election.cc:288] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Requested vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:18.244518 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66b932d884e4df8a78841103d88ded0" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:18.244553 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66b932d884e4df8a78841103d88ded0" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:18.244621 7182 raft_consensus.cc:3037] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.244665 6917 raft_consensus.cc:3037] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.245478 6917 raft_consensus.cc:2454] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 1.
I20240502 01:59:18.245527 7182 raft_consensus.cc:2454] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 1.
I20240502 01:59:18.245848 6721 leader_election.cc:302] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:18.246068 7789 raft_consensus.cc:2781] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:18.246274 7789 raft_consensus.cc:686] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Becoming Leader. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:18.246608 7789 consensus_queue.cc:237] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:18.249369 6641 catalog_manager.cc:5484] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: term changed from 0 to 1, leader changed from <none> to 85e78945910a4e1abfa101d2edc3644e (127.27.111.193). New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
W20240502 01:59:18.278357 6832 tablet.cc:2369] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:59:18.349550 7229 tablet.cc:2369] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:59:18.413475 7097 tablet.cc:2369] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:59:18.430759 7792 raft_consensus.cc:482] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:18.430996 7792 raft_consensus.cc:504] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.432111 7792 leader_election.cc:288] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:18.436720 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69d6e907b3b400f93721981732c006b" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:18.436884 6917 raft_consensus.cc:2454] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:18.437218 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69d6e907b3b400f93721981732c006b" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:18.437255 6985 leader_election.cc:302] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6101231591a458384f4ea6d1f0e0ae5, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:18.437426 6785 raft_consensus.cc:2454] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:18.437489 7792 raft_consensus.cc:2781] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:18.437569 7792 raft_consensus.cc:482] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:18.437619 7792 raft_consensus.cc:3037] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.438764 7792 raft_consensus.cc:504] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.439078 7792 leader_election.cc:288] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Requested vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:18.439360 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69d6e907b3b400f93721981732c006b" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:18.439392 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69d6e907b3b400f93721981732c006b" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:18.439461 6785 raft_consensus.cc:3037] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.439488 6917 raft_consensus.cc:3037] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.440204 6917 raft_consensus.cc:2454] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:18.440271 6785 raft_consensus.cc:2454] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:18.440474 6985 leader_election.cc:302] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6101231591a458384f4ea6d1f0e0ae5, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:18.440639 7792 raft_consensus.cc:2781] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:18.440845 7792 raft_consensus.cc:686] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Becoming Leader. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Running, Role: LEADER
I20240502 01:59:18.441143 7792 consensus_queue.cc:237] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.443856 6641 catalog_manager.cc:5484] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 01:59:18.555807 7787 raft_consensus.cc:482] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:18.555992 7787 raft_consensus.cc:504] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.557188 7787 leader_election.cc:288] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:18.561010 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc5daef41bc44c4855f2bcb44e1c6d5" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:18.561168 7182 raft_consensus.cc:2454] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 0.
I20240502 01:59:18.561213 7050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc5daef41bc44c4855f2bcb44e1c6d5" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 01:59:18.561415 7050 raft_consensus.cc:2454] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 0.
I20240502 01:59:18.561535 6851 leader_election.cc:302] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:18.561771 7787 raft_consensus.cc:2781] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:18.561828 7787 raft_consensus.cc:482] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:18.561856 7787 raft_consensus.cc:3037] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.562696 7787 raft_consensus.cc:504] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.563053 7787 leader_election.cc:288] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 election: Requested vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:18.563338 7050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc5daef41bc44c4855f2bcb44e1c6d5" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:18.563339 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc5daef41bc44c4855f2bcb44e1c6d5" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:18.563446 7050 raft_consensus.cc:3037] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.563447 7182 raft_consensus.cc:3037] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.564379 7050 raft_consensus.cc:2454] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 1.
I20240502 01:59:18.564410 7182 raft_consensus.cc:2454] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 1.
I20240502 01:59:18.564653 6852 leader_election.cc:302] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6101231591a458384f4ea6d1f0e0ae5, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:18.564844 7787 raft_consensus.cc:2781] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:18.565043 7787 raft_consensus.cc:686] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Becoming Leader. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Running, Role: LEADER
I20240502 01:59:18.565351 7787 consensus_queue.cc:237] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.568004 6641 catalog_manager.cc:5484] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
W20240502 01:59:18.728163 7807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:18.728411 7807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:18.731331 7807 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 01:59:18.783576 7830 consensus_queue.cc:1035] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:18.789122 7835 consensus_queue.cc:1035] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:18.822501 7182 raft_consensus.cc:1261] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:18.822505 7050 raft_consensus.cc:1261] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:18.822979 7787 consensus_queue.cc:1035] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:18.823302 7805 consensus_queue.cc:1035] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:18.824401 6785 raft_consensus.cc:1261] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:18.825054 6917 raft_consensus.cc:1261] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:18.825579 7792 consensus_queue.cc:1035] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:18.827836 7801 consensus_queue.cc:1035] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:18.836622 7849 mvcc.cc:205] Tried to move back new op lower bound from 7023063690530533376 to 7023063689485856768. Current Snapshot: MvccSnapshot[applied={T|T < 7023063690530533376}]
I20240502 01:59:18.849558 7834 mvcc.cc:205] Tried to move back new op lower bound from 7023063690623287296 to 7023063689485856768. Current Snapshot: MvccSnapshot[applied={T|T < 7023063690623287296}]
I20240502 01:59:18.849762 7833 mvcc.cc:205] Tried to move back new op lower bound from 7023063690539175936 to 7023063688977186816. Current Snapshot: MvccSnapshot[applied={T|T < 7023063690539175936}]
I20240502 01:59:18.851800 7854 mvcc.cc:205] Tried to move back new op lower bound from 7023063690530533376 to 7023063689485856768. Current Snapshot: MvccSnapshot[applied={T|T < 7023063690530533376}]
INSERT report
rows total: 2000
time total: 115.688 ms
time per row: 0.0578438 ms
I20240502 01:59:18.888217 6641 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32904:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:18.893050 6764 tablet_service.cc:1467] Processing CreateTablet for tablet c8c237c50773494abf8813494be0f184 (DEFAULT_TABLE table=rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.893221 7161 tablet_service.cc:1467] Processing CreateTablet for tablet c8c237c50773494abf8813494be0f184 (DEFAULT_TABLE table=rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.893389 6764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c237c50773494abf8813494be0f184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.893360 6765 tablet_service.cc:1467] Processing CreateTablet for tablet 47a276e9780e448a9b3c6eab67925514 (DEFAULT_TABLE table=rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.893492 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c237c50773494abf8813494be0f184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.893637 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a276e9780e448a9b3c6eab67925514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.894150 7029 tablet_service.cc:1467] Processing CreateTablet for tablet c8c237c50773494abf8813494be0f184 (DEFAULT_TABLE table=rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.894448 7029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c237c50773494abf8813494be0f184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.895905 7030 tablet_service.cc:1467] Processing CreateTablet for tablet 47a276e9780e448a9b3c6eab67925514 (DEFAULT_TABLE table=rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.896127 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a276e9780e448a9b3c6eab67925514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.896340 6894 tablet_service.cc:1467] Processing CreateTablet for tablet 0359dfbfd11746f1bf62a465f14634ec (DEFAULT_TABLE table=rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.896443 6765 tablet_service.cc:1467] Processing CreateTablet for tablet 0359dfbfd11746f1bf62a465f14634ec (DEFAULT_TABLE table=rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.896587 6894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0359dfbfd11746f1bf62a465f14634ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.894166 7028 tablet_service.cc:1467] Processing CreateTablet for tablet 0359dfbfd11746f1bf62a465f14634ec (DEFAULT_TABLE table=rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.896667 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0359dfbfd11746f1bf62a465f14634ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.896800 7028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0359dfbfd11746f1bf62a465f14634ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.898624 7872 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:18.898888 7874 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:18.899142 7161 tablet_service.cc:1467] Processing CreateTablet for tablet 47a276e9780e448a9b3c6eab67925514 (DEFAULT_TABLE table=rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:18.899344 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a276e9780e448a9b3c6eab67925514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:18.899518 7873 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:18.899626 7872 tablet_bootstrap.cc:654] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.899681 7874 tablet_bootstrap.cc:654] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.900425 7873 tablet_bootstrap.cc:654] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.900635 7871 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:18.900934 7874 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:18.901032 7874 ts_tablet_manager.cc:1367] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:18.901511 7874 raft_consensus.cc:348] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.901633 7874 raft_consensus.cc:374] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.901609 7871 tablet_bootstrap.cc:654] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.901680 7874 raft_consensus.cc:729] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.901847 7874 consensus_queue.cc:260] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.902132 7872 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:18.902207 7872 ts_tablet_manager.cc:1367] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:59:18.902280 7874 ts_tablet_manager.cc:1398] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:18.902669 7873 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:18.902762 7873 ts_tablet_manager.cc:1367] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:18.902751 7872 raft_consensus.cc:348] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:18.902841 7872 raft_consensus.cc:374] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.902868 7872 raft_consensus.cc:729] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.902982 7872 consensus_queue.cc:260] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:18.903230 7872 ts_tablet_manager.cc:1398] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:18.903219 7873 raft_consensus.cc:348] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.903309 7871 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:18.903306 7873 raft_consensus.cc:374] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.903342 7873 raft_consensus.cc:729] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.903380 7871 ts_tablet_manager.cc:1367] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:18.903412 7872 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:18.903455 7873 consensus_queue.cc:260] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.903718 7873 ts_tablet_manager.cc:1398] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:18.903879 7873 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:18.904034 7871 raft_consensus.cc:348] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.904121 7871 raft_consensus.cc:374] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.904151 7871 raft_consensus.cc:729] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.904263 7872 tablet_bootstrap.cc:654] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.904289 7871 consensus_queue.cc:260] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.904556 7871 ts_tablet_manager.cc:1398] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:18.904722 7871 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:18.904747 7873 tablet_bootstrap.cc:654] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.905586 7871 tablet_bootstrap.cc:654] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.906302 7872 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:18.906446 7872 ts_tablet_manager.cc:1367] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:18.906884 7872 raft_consensus.cc:348] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.907002 7872 raft_consensus.cc:374] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.907032 7872 raft_consensus.cc:729] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.907122 7873 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:18.907142 7872 consensus_queue.cc:260] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.907210 7873 ts_tablet_manager.cc:1367] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:59:18.907424 7872 ts_tablet_manager.cc:1398] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:18.907577 7872 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:18.907605 7871 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:18.907578 7873 raft_consensus.cc:348] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:18.907668 7871 ts_tablet_manager.cc:1367] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:18.907671 7873 raft_consensus.cc:374] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.907706 7873 raft_consensus.cc:729] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.907829 7873 consensus_queue.cc:260] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:18.908118 7871 raft_consensus.cc:348] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:18.908239 7871 raft_consensus.cc:374] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.908269 7871 raft_consensus.cc:729] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.908390 7871 consensus_queue.cc:260] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:18.908653 7872 tablet_bootstrap.cc:654] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.908655 7871 ts_tablet_manager.cc:1398] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:18.908941 7873 ts_tablet_manager.cc:1398] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:59:18.909121 7873 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:18.910166 7872 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:18.910243 7872 ts_tablet_manager.cc:1367] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:18.910596 7873 tablet_bootstrap.cc:654] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:18.910661 7872 raft_consensus.cc:348] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.910778 7872 raft_consensus.cc:374] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.910818 7872 raft_consensus.cc:729] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.910941 7872 consensus_queue.cc:260] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.911290 7872 ts_tablet_manager.cc:1398] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:18.912009 7873 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:18.912088 7873 ts_tablet_manager.cc:1367] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:18.912509 7873 raft_consensus.cc:348] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.912602 7873 raft_consensus.cc:374] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:18.912638 7873 raft_consensus.cc:729] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:18.912755 7873 consensus_queue.cc:260] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.913009 7873 ts_tablet_manager.cc:1398] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:18.924165 7830 raft_consensus.cc:482] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:18.924247 7830 raft_consensus.cc:504] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.924607 7830 leader_election.cc:288] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:18.925001 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0359dfbfd11746f1bf62a465f14634ec" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:18.925143 6916 raft_consensus.cc:2454] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:18.925447 6721 leader_election.cc:302] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:18.925642 7830 raft_consensus.cc:2781] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:18.925709 7830 raft_consensus.cc:482] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:18.925743 7830 raft_consensus.cc:3037] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.926690 7830 raft_consensus.cc:504] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.926997 7830 leader_election.cc:288] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Requested vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:18.927243 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0359dfbfd11746f1bf62a465f14634ec" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:18.927347 6916 raft_consensus.cc:3037] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.928242 6916 raft_consensus.cc:2454] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 1.
I20240502 01:59:18.928493 6721 leader_election.cc:302] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:18.928740 7050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0359dfbfd11746f1bf62a465f14634ec" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 01:59:18.928766 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0359dfbfd11746f1bf62a465f14634ec" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:18.928862 7050 raft_consensus.cc:2454] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:18.929031 7830 raft_consensus.cc:2781] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:18.929121 7830 raft_consensus.cc:686] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Becoming Leader. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:18.929255 7830 consensus_queue.cc:237] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.930788 6641 catalog_manager.cc:5484] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e reported cstate change: term changed from 0 to 1, leader changed from <none> to 85e78945910a4e1abfa101d2edc3644e (127.27.111.193). New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 01:59:18.941550 7801 raft_consensus.cc:482] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:18.941646 7801 raft_consensus.cc:504] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:18.942046 7801 leader_election.cc:288] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:18.942363 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a276e9780e448a9b3c6eab67925514" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:18.942519 6785 raft_consensus.cc:2454] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:18.942797 6984 leader_election.cc:302] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:18.942955 7801 raft_consensus.cc:2781] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:18.943030 7801 raft_consensus.cc:482] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:18.943076 7801 raft_consensus.cc:3037] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.944021 7801 raft_consensus.cc:504] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:18.944334 7801 leader_election.cc:288] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Requested vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:18.944639 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a276e9780e448a9b3c6eab67925514" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:18.944734 6785 raft_consensus.cc:3037] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.945624 6785 raft_consensus.cc:2454] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:18.945881 6984 leader_election.cc:302] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:18.945900 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a276e9780e448a9b3c6eab67925514" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:18.945992 7181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a276e9780e448a9b3c6eab67925514" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:18.946051 7801 raft_consensus.cc:2781] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:18.946012 7182 raft_consensus.cc:2454] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:18.946111 7801 raft_consensus.cc:686] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Becoming Leader. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Running, Role: LEADER
I20240502 01:59:18.946113 7181 raft_consensus.cc:3037] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.946241 7801 consensus_queue.cc:237] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:18.947530 7181 raft_consensus.cc:2454] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:18.947882 6641 catalog_manager.cc:5484] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:18.960525 7832 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:18.960613 7832 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.960932 7832 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:18.961247 7181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c237c50773494abf8813494be0f184" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:18.961283 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c237c50773494abf8813494be0f184" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 01:59:18.961382 7181 raft_consensus.cc:2454] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:18.961381 7049 raft_consensus.cc:2454] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:18.961643 6721 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:18.961793 7832 raft_consensus.cc:2781] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:18.961854 7832 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:18.961913 7832 raft_consensus.cc:3037] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.962711 7832 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.963043 7832 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Requested vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:18.963291 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c237c50773494abf8813494be0f184" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:18.963291 7181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c237c50773494abf8813494be0f184" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:18.963382 7049 raft_consensus.cc:3037] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.963382 7181 raft_consensus.cc:3037] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:18.964237 7181 raft_consensus.cc:2454] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 1.
I20240502 01:59:18.964265 7049 raft_consensus.cc:2454] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 1.
I20240502 01:59:18.964490 6721 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:18.964643 7832 raft_consensus.cc:2781] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:18.964720 7832 raft_consensus.cc:686] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Becoming Leader. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:18.964854 7832 consensus_queue.cc:237] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:18.966298 6641 catalog_manager.cc:5484] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: term changed from 0 to 1, leader changed from <none> to 85e78945910a4e1abfa101d2edc3644e (127.27.111.193). New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
W20240502 01:59:19.081326 7889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:19.081563 7889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:19.084583 7889 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 01:59:19.176290 7049 raft_consensus.cc:3037] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:19.176276 6916 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:19.176378 7050 raft_consensus.cc:1261] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:19.177017 7832 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:19.177469 7049 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:19.177501 6785 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:19.177865 7792 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:19.178002 7181 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:19.178375 7182 raft_consensus.cc:1261] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:19.179016 7835 consensus_queue.cc:1035] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:19.179194 7830 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:19.179472 7830 consensus_queue.cc:1035] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:19.179850 7792 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.901 ms
time per row: 0.0574506 ms
I20240502 01:59:19.240981 6637 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32904:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:19.246608 6765 tablet_service.cc:1467] Processing CreateTablet for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.246789 7161 tablet_service.cc:1467] Processing CreateTablet for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.246969 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.246898 6894 tablet_service.cc:1467] Processing CreateTablet for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.247040 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.246898 6896 tablet_service.cc:1467] Processing CreateTablet for tablet 17235e019e644b258266636fb6be6648 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.247197 6894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.247843 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17235e019e644b258266636fb6be6648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.249197 7871 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:19.249369 7874 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:19.249922 7871 tablet_bootstrap.cc:654] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.250094 7874 tablet_bootstrap.cc:654] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.250517 7872 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:19.251122 7874 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:19.251204 7874 ts_tablet_manager.cc:1367] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:19.251236 7161 tablet_service.cc:1467] Processing CreateTablet for tablet 17235e019e644b258266636fb6be6648 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.251413 7872 tablet_bootstrap.cc:654] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.251482 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17235e019e644b258266636fb6be6648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.251699 7874 raft_consensus.cc:348] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.251816 7874 raft_consensus.cc:374] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.251849 7874 raft_consensus.cc:729] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.251965 7874 consensus_queue.cc:260] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.252130 6896 tablet_service.cc:1467] Processing CreateTablet for tablet 8529a93abd6e4d62bbc0bddcf2fda78a (DEFAULT_TABLE table=rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.252214 7874 ts_tablet_manager.cc:1398] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:19.252382 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8529a93abd6e4d62bbc0bddcf2fda78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.253480 7871 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:19.253562 7871 ts_tablet_manager.cc:1367] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:59:19.254042 7871 raft_consensus.cc:348] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.254141 7871 raft_consensus.cc:374] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.254175 7871 raft_consensus.cc:729] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.254292 7871 consensus_queue.cc:260] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.254340 7160 tablet_service.cc:1467] Processing CreateTablet for tablet 8529a93abd6e4d62bbc0bddcf2fda78a (DEFAULT_TABLE table=rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.254587 7871 ts_tablet_manager.cc:1398] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:19.254596 7160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8529a93abd6e4d62bbc0bddcf2fda78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.254770 7871 tablet_bootstrap.cc:492] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:19.254868 7028 tablet_service.cc:1467] Processing CreateTablet for tablet 17235e019e644b258266636fb6be6648 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.255131 7028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17235e019e644b258266636fb6be6648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.255511 7871 tablet_bootstrap.cc:654] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.257233 7874 tablet_bootstrap.cc:492] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:19.257299 6765 tablet_service.cc:1467] Processing CreateTablet for tablet 8529a93abd6e4d62bbc0bddcf2fda78a (DEFAULT_TABLE table=rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.257715 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8529a93abd6e4d62bbc0bddcf2fda78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.257954 7874 tablet_bootstrap.cc:654] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.258044 7873 tablet_bootstrap.cc:492] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:19.258862 7873 tablet_bootstrap.cc:654] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.259276 7874 tablet_bootstrap.cc:492] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:19.259410 7874 ts_tablet_manager.cc:1367] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:19.259837 7872 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:19.259936 7872 ts_tablet_manager.cc:1367] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:59:19.260095 7871 tablet_bootstrap.cc:492] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:19.260221 7871 ts_tablet_manager.cc:1367] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:19.260388 7872 raft_consensus.cc:348] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.260497 7872 raft_consensus.cc:374] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.260537 7872 raft_consensus.cc:729] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.260658 7872 consensus_queue.cc:260] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.260692 7871 raft_consensus.cc:348] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:19.260681 7874 raft_consensus.cc:348] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:19.260774 7871 raft_consensus.cc:374] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.260785 7874 raft_consensus.cc:374] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.260810 7871 raft_consensus.cc:729] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.260815 7874 raft_consensus.cc:729] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.260963 7872 ts_tablet_manager.cc:1398] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:19.260932 7874 consensus_queue.cc:260] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:19.261229 7871 consensus_queue.cc:260] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:19.261332 7874 ts_tablet_manager.cc:1398] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:19.261636 7871 ts_tablet_manager.cc:1398] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:19.261644 7873 tablet_bootstrap.cc:492] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:19.261713 7873 ts_tablet_manager.cc:1367] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:59:19.261790 7871 tablet_bootstrap.cc:492] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:19.262178 7873 raft_consensus.cc:348] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:19.262286 7873 raft_consensus.cc:374] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.262301 7872 tablet_bootstrap.cc:492] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:19.262317 7873 raft_consensus.cc:729] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.262487 7873 consensus_queue.cc:260] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:19.262768 7873 ts_tablet_manager.cc:1398] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:19.263238 7872 tablet_bootstrap.cc:654] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.263578 7871 tablet_bootstrap.cc:654] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.264364 7874 tablet_bootstrap.cc:492] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:19.265107 7872 tablet_bootstrap.cc:492] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:19.265197 7872 ts_tablet_manager.cc:1367] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:19.265413 7874 tablet_bootstrap.cc:654] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.265619 7872 raft_consensus.cc:348] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:19.265731 7872 raft_consensus.cc:374] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.265771 7872 raft_consensus.cc:729] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.265911 7872 consensus_queue.cc:260] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:19.266196 7872 ts_tablet_manager.cc:1398] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:19.267058 7871 tablet_bootstrap.cc:492] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:19.267153 7871 ts_tablet_manager.cc:1367] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:59:19.267589 7871 raft_consensus.cc:348] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:19.267696 7871 raft_consensus.cc:374] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.267738 7871 raft_consensus.cc:729] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.267861 7871 consensus_queue.cc:260] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:19.268026 7874 tablet_bootstrap.cc:492] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:19.268102 7874 ts_tablet_manager.cc:1367] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:59:19.268133 7871 ts_tablet_manager.cc:1398] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:19.268500 7874 raft_consensus.cc:348] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:19.268599 7874 raft_consensus.cc:374] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.268640 7874 raft_consensus.cc:729] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.268748 7874 consensus_queue.cc:260] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:19.269047 7874 ts_tablet_manager.cc:1398] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:19.403126 7787 raft_consensus.cc:482] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:19.403254 7787 raft_consensus.cc:504] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:19.403672 7787 leader_election.cc:288] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:19.404017 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17235e019e644b258266636fb6be6648" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:19.404073 7050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17235e019e644b258266636fb6be6648" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 01:59:19.404174 7182 raft_consensus.cc:2454] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 0.
I20240502 01:59:19.404191 7050 raft_consensus.cc:2454] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 0.
I20240502 01:59:19.404450 6852 leader_election.cc:302] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6101231591a458384f4ea6d1f0e0ae5, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:19.404608 7787 raft_consensus.cc:2781] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:19.404665 7787 raft_consensus.cc:482] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:19.404701 7787 raft_consensus.cc:3037] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:19.405469 7787 raft_consensus.cc:504] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:19.405754 7787 leader_election.cc:288] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 election: Requested vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:19.406060 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17235e019e644b258266636fb6be6648" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:19.406108 7050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17235e019e644b258266636fb6be6648" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:19.406159 7182 raft_consensus.cc:3037] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:19.406229 7050 raft_consensus.cc:3037] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:19.406930 7182 raft_consensus.cc:2454] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 1.
I20240502 01:59:19.407128 7050 raft_consensus.cc:2454] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 1.
I20240502 01:59:19.407196 6851 leader_election.cc:302] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:19.407362 7787 raft_consensus.cc:2781] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:19.407512 7787 raft_consensus.cc:686] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Becoming Leader. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Running, Role: LEADER
I20240502 01:59:19.407637 7787 consensus_queue.cc:237] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:19.409101 6641 catalog_manager.cc:5484] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 01:59:19.413487 7836 raft_consensus.cc:482] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:19.413596 7836 raft_consensus.cc:504] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:19.414821 7836 leader_election.cc:288] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:19.419569 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8529a93abd6e4d62bbc0bddcf2fda78a" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:19.419564 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8529a93abd6e4d62bbc0bddcf2fda78a" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:19.419699 6916 raft_consensus.cc:2454] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 0.
I20240502 01:59:19.419710 6785 raft_consensus.cc:2454] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 0.
I20240502 01:59:19.420040 7117 leader_election.cc:302] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:19.420284 7836 raft_consensus.cc:2781] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:19.420360 7836 raft_consensus.cc:482] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:19.420416 7836 raft_consensus.cc:3037] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:19.421237 7836 raft_consensus.cc:504] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:19.421622 7836 leader_election.cc:288] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 election: Requested vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:19.421981 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8529a93abd6e4d62bbc0bddcf2fda78a" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:19.421917 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8529a93abd6e4d62bbc0bddcf2fda78a" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:19.422101 6916 raft_consensus.cc:3037] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:19.422086 6785 raft_consensus.cc:3037] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:19.423005 6916 raft_consensus.cc:2454] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 1.
I20240502 01:59:19.423107 6785 raft_consensus.cc:2454] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 1.
I20240502 01:59:19.423290 7117 leader_election.cc:302] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:19.423450 7836 raft_consensus.cc:2781] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:19.423635 7836 raft_consensus.cc:686] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Becoming Leader. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Running, Role: LEADER
I20240502 01:59:19.423964 7836 consensus_queue.cc:237] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:19.426287 6641 catalog_manager.cc:5484] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 86bc8001c3464b929cd5c873861baa0c (127.27.111.196). New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:19.619645 7787 raft_consensus.cc:482] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:19.619757 7787 raft_consensus.cc:504] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.620128 7787 leader_election.cc:288] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:19.620926 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecbf1c8ee8c46b195d2374a5db81ac0" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:19.621096 7182 raft_consensus.cc:2454] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 0.
I20240502 01:59:19.621412 6851 leader_election.cc:302] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:19.621606 7787 raft_consensus.cc:2781] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:19.621671 7787 raft_consensus.cc:482] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:19.621699 7787 raft_consensus.cc:3037] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:19.622610 7787 raft_consensus.cc:504] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.622980 7787 leader_election.cc:288] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 election: Requested vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:19.623306 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecbf1c8ee8c46b195d2374a5db81ac0" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:19.623430 7182 raft_consensus.cc:3037] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:19.624166 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecbf1c8ee8c46b195d2374a5db81ac0" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:19.624289 6785 raft_consensus.cc:2454] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 0.
I20240502 01:59:19.624267 6784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ecbf1c8ee8c46b195d2374a5db81ac0" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:19.624375 7182 raft_consensus.cc:2454] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 1.
I20240502 01:59:19.624439 6784 raft_consensus.cc:3037] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:19.624644 6851 leader_election.cc:302] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:19.624886 7787 raft_consensus.cc:2781] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:19.624976 7787 raft_consensus.cc:686] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Becoming Leader. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Running, Role: LEADER
I20240502 01:59:19.625118 7787 consensus_queue.cc:237] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.625310 6784 raft_consensus.cc:2454] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 1.
I20240502 01:59:19.626856 6641 catalog_manager.cc:5484] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:19.795560 7932 consensus_queue.cc:1035] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:19.799310 7787 consensus_queue.cc:1035] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:19.810424 7941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:19.810680 7941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:19.813676 7941 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 01:59:19.861780 7836 consensus_queue.cc:1035] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:19.865707 7937 consensus_queue.cc:1035] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:19.890437 7182 raft_consensus.cc:1261] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:19.890653 6784 raft_consensus.cc:1261] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:19.893666 7787 consensus_queue.cc:1035] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:19.894028 7787 consensus_queue.cc:1035] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 95.6138 ms
time per row: 0.0478069 ms
I20240502 01:59:19.949193 6641 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32904:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:19.954555 6896 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6c07e3923d47f29ef32561d7f438e1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.954545 7028 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6c07e3923d47f29ef32561d7f438e1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.954865 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f6c07e3923d47f29ef32561d7f438e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.954865 7028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f6c07e3923d47f29ef32561d7f438e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.955168 7030 tablet_service.cc:1467] Processing CreateTablet for tablet d4d429fe60e74108bc0c9ca8509365f2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.955435 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d429fe60e74108bc0c9ca8509365f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.955793 6764 tablet_service.cc:1467] Processing CreateTablet for tablet e6ac26fa6de44f7ca68e6c0b6b959db7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.956049 6764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ac26fa6de44f7ca68e6c0b6b959db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.957319 6765 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6c07e3923d47f29ef32561d7f438e1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.957569 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f6c07e3923d47f29ef32561d7f438e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.958348 6896 tablet_service.cc:1467] Processing CreateTablet for tablet d4d429fe60e74108bc0c9ca8509365f2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.958318 7160 tablet_service.cc:1467] Processing CreateTablet for tablet d4d429fe60e74108bc0c9ca8509365f2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.958606 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d429fe60e74108bc0c9ca8509365f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.958638 7160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d429fe60e74108bc0c9ca8509365f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.958851 8002 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:19.959774 8002 tablet_bootstrap.cc:654] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.959754 7029 tablet_service.cc:1467] Processing CreateTablet for tablet e6ac26fa6de44f7ca68e6c0b6b959db7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.959980 7029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ac26fa6de44f7ca68e6c0b6b959db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.958634 7161 tablet_service.cc:1467] Processing CreateTablet for tablet e6ac26fa6de44f7ca68e6c0b6b959db7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:19.960867 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ac26fa6de44f7ca68e6c0b6b959db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:19.961234 8004 tablet_bootstrap.cc:492] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:19.961993 8002 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:19.962054 8004 tablet_bootstrap.cc:654] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.962090 8002 ts_tablet_manager.cc:1367] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:19.962908 8002 raft_consensus.cc:348] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.963047 8002 raft_consensus.cc:374] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.963099 8002 raft_consensus.cc:729] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.963140 8004 tablet_bootstrap.cc:492] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:19.963212 8004 ts_tablet_manager.cc:1367] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:19.963232 8002 consensus_queue.cc:260] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.963529 8002 ts_tablet_manager.cc:1398] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:19.963627 8004 raft_consensus.cc:348] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:19.963718 8002 tablet_bootstrap.cc:492] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:19.963764 8004 raft_consensus.cc:374] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.963805 8004 raft_consensus.cc:729] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.963953 8004 consensus_queue.cc:260] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:19.964015 8001 tablet_bootstrap.cc:492] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:19.964219 8004 ts_tablet_manager.cc:1398] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:19.964376 8004 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:19.964613 8002 tablet_bootstrap.cc:654] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.964944 7832 raft_consensus.cc:482] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:19.965034 7832 raft_consensus.cc:504] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.965150 8004 tablet_bootstrap.cc:654] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.963337 8000 tablet_bootstrap.cc:492] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:19.965385 8001 tablet_bootstrap.cc:654] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.965451 7832 leader_election.cc:288] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:19.966068 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 01:59:19.966181 8000 tablet_bootstrap.cc:654] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.966578 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:19.966789 8004 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:19.966840 8001 tablet_bootstrap.cc:492] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:19.966862 8004 ts_tablet_manager.cc:1367] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:19.966905 8001 ts_tablet_manager.cc:1367] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:19.967262 8002 tablet_bootstrap.cc:492] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:19.967264 8004 raft_consensus.cc:348] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.967264 8001 raft_consensus.cc:348] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:19.967339 8002 ts_tablet_manager.cc:1367] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:59:19.967352 8004 raft_consensus.cc:374] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.967352 8001 raft_consensus.cc:374] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.967387 8004 raft_consensus.cc:729] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.967399 8001 raft_consensus.cc:729] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.967522 8004 consensus_queue.cc:260] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.967515 8001 consensus_queue.cc:260] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:19.967787 8001 ts_tablet_manager.cc:1398] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:19.967749 8002 raft_consensus.cc:348] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:19.967849 8002 raft_consensus.cc:374] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.967902 8002 raft_consensus.cc:729] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.967937 8001 tablet_bootstrap.cc:492] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:19.967949 8004 ts_tablet_manager.cc:1398] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:19.968014 8002 consensus_queue.cc:260] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
W20240502 01:59:19.968060 6720 leader_election.cc:341] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:59:19.968322 8002 ts_tablet_manager.cc:1398] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:19.968688 8001 tablet_bootstrap.cc:654] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
W20240502 01:59:19.968977 6721 leader_election.cc:341] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:59:19.969056 6721 leader_election.cc:302] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e; no voters: 86bc8001c3464b929cd5c873861baa0c, c6101231591a458384f4ea6d1f0e0ae5
I20240502 01:59:19.969579 7832 raft_consensus.cc:2726] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:59:19.969596 8001 tablet_bootstrap.cc:492] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:19.969628 8000 tablet_bootstrap.cc:492] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:19.969673 8001 ts_tablet_manager.cc:1367] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:19.969713 8000 ts_tablet_manager.cc:1367] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:59:19.970038 8001 raft_consensus.cc:348] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:19.970155 8001 raft_consensus.cc:374] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.970213 8001 raft_consensus.cc:729] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.970171 8000 raft_consensus.cc:348] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:19.970271 8000 raft_consensus.cc:374] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.970310 8000 raft_consensus.cc:729] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.970332 8001 consensus_queue.cc:260] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:19.970651 8001 ts_tablet_manager.cc:1398] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:19.970796 8001 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:19.971203 8000 consensus_queue.cc:260] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:19.971468 8000 ts_tablet_manager.cc:1398] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:59:19.971588 8001 tablet_bootstrap.cc:654] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.971629 8000 tablet_bootstrap.cc:492] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:19.972523 8000 tablet_bootstrap.cc:654] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:19.973417 8001 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:19.973501 8001 ts_tablet_manager.cc:1367] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:19.974017 8001 raft_consensus.cc:348] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.974124 8001 raft_consensus.cc:374] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.974157 8001 raft_consensus.cc:729] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.974170 8000 tablet_bootstrap.cc:492] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:19.974275 8001 consensus_queue.cc:260] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:19.974357 8000 ts_tablet_manager.cc:1367] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:59:19.974571 8001 ts_tablet_manager.cc:1398] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:19.974810 8000 raft_consensus.cc:348] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:19.974921 8000 raft_consensus.cc:374] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:19.974969 8000 raft_consensus.cc:729] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:19.975080 8000 consensus_queue.cc:260] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:19.975353 8000 ts_tablet_manager.cc:1398] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:20.013303 7836 raft_consensus.cc:482] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:20.013440 7836 raft_consensus.cc:504] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.013926 7836 leader_election.cc:288] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:20.014428 6784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:20.014580 6784 raft_consensus.cc:2454] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 0.
I20240502 01:59:20.015421 7116 leader_election.cc:302] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, 86bc8001c3464b929cd5c873861baa0c; no voters:
I20240502 01:59:20.015686 7836 raft_consensus.cc:2781] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:20.015759 7836 raft_consensus.cc:482] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:20.015798 7836 raft_consensus.cc:3037] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.016772 7836 raft_consensus.cc:504] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.017138 7836 leader_election.cc:288] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 election: Requested vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:20.017462 6784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:20.017593 6784 raft_consensus.cc:3037] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.018558 6784 raft_consensus.cc:2454] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 1.
I20240502 01:59:20.018823 7116 leader_election.cc:302] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, 86bc8001c3464b929cd5c873861baa0c; no voters:
I20240502 01:59:20.018994 7836 raft_consensus.cc:2781] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:20.019011 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 01:59:20.019079 7050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:20.019107 7836 raft_consensus.cc:686] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Becoming Leader. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Running, Role: LEADER
I20240502 01:59:20.019148 7049 raft_consensus.cc:2454] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 0.
I20240502 01:59:20.019256 7836 consensus_queue.cc:237] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.020848 6643 catalog_manager.cc:5484] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 86bc8001c3464b929cd5c873861baa0c (127.27.111.196). New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:20.064313 7832 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:20.064452 7832 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:20.064874 7832 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:20.065315 7050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6c07e3923d47f29ef32561d7f438e1" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 01:59:20.065377 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6c07e3923d47f29ef32561d7f438e1" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:20.065447 7050 raft_consensus.cc:2454] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:20.065542 6916 raft_consensus.cc:2454] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:20.065778 6721 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:20.065946 7832 raft_consensus.cc:2781] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:20.066009 7832 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:20.066057 7832 raft_consensus.cc:3037] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.066920 7832 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:20.067245 7832 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Requested vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:20.067548 7050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6c07e3923d47f29ef32561d7f438e1" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:20.067586 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6c07e3923d47f29ef32561d7f438e1" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:20.067646 7050 raft_consensus.cc:3037] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.067701 6916 raft_consensus.cc:3037] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.068364 7050 raft_consensus.cc:2454] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 1.
I20240502 01:59:20.068473 6916 raft_consensus.cc:2454] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 1.
I20240502 01:59:20.068619 6721 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:20.068773 7832 raft_consensus.cc:2781] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:20.068857 7832 raft_consensus.cc:686] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Becoming Leader. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:20.069008 7832 consensus_queue.cc:237] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:20.070700 6643 catalog_manager.cc:5484] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: term changed from 0 to 1, leader changed from <none> to 85e78945910a4e1abfa101d2edc3644e (127.27.111.193). New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:20.145849 7792 raft_consensus.cc:482] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:20.145992 7792 raft_consensus.cc:504] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:20.146356 7792 leader_election.cc:288] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:20.146765 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d429fe60e74108bc0c9ca8509365f2" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:20.146776 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d429fe60e74108bc0c9ca8509365f2" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:20.146896 6916 raft_consensus.cc:2454] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:20.146908 7182 raft_consensus.cc:2454] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:20.147192 6985 leader_election.cc:302] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6101231591a458384f4ea6d1f0e0ae5, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:20.147400 7792 raft_consensus.cc:2781] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:20.147466 7792 raft_consensus.cc:482] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:20.147501 7792 raft_consensus.cc:3037] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.148461 7792 raft_consensus.cc:504] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:20.148794 7792 leader_election.cc:288] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Requested vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:20.149084 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d429fe60e74108bc0c9ca8509365f2" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:20.149102 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d429fe60e74108bc0c9ca8509365f2" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:20.149180 7182 raft_consensus.cc:3037] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.149201 6916 raft_consensus.cc:3037] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.149906 7182 raft_consensus.cc:2454] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:20.149919 6916 raft_consensus.cc:2454] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:20.150156 6985 leader_election.cc:302] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6101231591a458384f4ea6d1f0e0ae5, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:20.150314 7792 raft_consensus.cc:2781] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:20.150454 7792 raft_consensus.cc:686] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Becoming Leader. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Running, Role: LEADER
I20240502 01:59:20.150589 7792 consensus_queue.cc:237] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:20.152071 6643 catalog_manager.cc:5484] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
W20240502 01:59:20.300415 8018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:20.300639 8018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:20.303512 8018 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 01:59:20.386123 6916 raft_consensus.cc:1261] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:20.386590 7947 consensus_queue.cc:1035] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:20.389850 6917 raft_consensus.cc:1261] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:20.390244 7835 consensus_queue.cc:1035] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:20.390893 7050 raft_consensus.cc:1261] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:20.391052 7049 raft_consensus.cc:3037] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.392098 7049 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:20.392565 6784 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:20.392874 7835 consensus_queue.cc:1035] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:20.392946 7836 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:20.393657 7836 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:20.394626 7182 raft_consensus.cc:1261] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:59:20.395114 7792 consensus_queue.cc:1035] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.195 ms
time per row: 0.0520974 ms
I20240502 01:59:20.452414 6636 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32904:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:20.459697 6765 tablet_service.cc:1467] Processing CreateTablet for tablet 44c55b199a24432e902cce9121fc0780 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f2b695d4b3e54fdfad81f019e9ab7b4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.459838 6896 tablet_service.cc:1467] Processing CreateTablet for tablet 44c55b199a24432e902cce9121fc0780 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f2b695d4b3e54fdfad81f019e9ab7b4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.460081 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c55b199a24432e902cce9121fc0780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.460085 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c55b199a24432e902cce9121fc0780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.460636 6764 tablet_service.cc:1467] Processing CreateTablet for tablet e0e05f96dbc542eb999c16dcc0620666 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f2b695d4b3e54fdfad81f019e9ab7b4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.460714 7161 tablet_service.cc:1467] Processing CreateTablet for tablet 44c55b199a24432e902cce9121fc0780 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f2b695d4b3e54fdfad81f019e9ab7b4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.460875 6764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e05f96dbc542eb999c16dcc0620666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.460996 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c55b199a24432e902cce9121fc0780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.462253 8002 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:20.462970 7160 tablet_service.cc:1467] Processing CreateTablet for tablet 818dbc9a8e054eeaa122e7919a311806 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f2b695d4b3e54fdfad81f019e9ab7b4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.463032 8002 tablet_bootstrap.cc:654] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.463204 7160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818dbc9a8e054eeaa122e7919a311806. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.463987 8002 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:20.464071 8002 ts_tablet_manager.cc:1367] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:20.464455 8002 raft_consensus.cc:348] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.464566 8002 raft_consensus.cc:374] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.464604 8002 raft_consensus.cc:729] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.464728 8002 consensus_queue.cc:260] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.465003 8000 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:20.465047 8002 ts_tablet_manager.cc:1398] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:20.465214 8002 tablet_bootstrap.cc:492] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:20.465831 8002 tablet_bootstrap.cc:654] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.465838 8000 tablet_bootstrap.cc:654] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.466322 8004 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:20.467200 8004 tablet_bootstrap.cc:654] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.467588 6894 tablet_service.cc:1467] Processing CreateTablet for tablet 818dbc9a8e054eeaa122e7919a311806 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f2b695d4b3e54fdfad81f019e9ab7b4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.467633 7029 tablet_service.cc:1467] Processing CreateTablet for tablet e0e05f96dbc542eb999c16dcc0620666 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f2b695d4b3e54fdfad81f019e9ab7b4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.467856 6894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818dbc9a8e054eeaa122e7919a311806. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.467917 7029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e05f96dbc542eb999c16dcc0620666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.467942 7030 tablet_service.cc:1467] Processing CreateTablet for tablet 818dbc9a8e054eeaa122e7919a311806 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f2b695d4b3e54fdfad81f019e9ab7b4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.468151 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818dbc9a8e054eeaa122e7919a311806. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.470645 8000 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:20.470736 8000 ts_tablet_manager.cc:1367] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240502 01:59:20.471024 8001 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:20.471154 8000 raft_consensus.cc:348] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.471251 8000 raft_consensus.cc:374] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.471289 8000 raft_consensus.cc:729] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.471421 8000 consensus_queue.cc:260] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.471680 8000 ts_tablet_manager.cc:1398] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:20.471832 8001 tablet_bootstrap.cc:654] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.471865 8000 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:20.472098 8002 tablet_bootstrap.cc:492] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:20.472193 8002 ts_tablet_manager.cc:1367] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:59:20.472654 8002 raft_consensus.cc:348] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:20.472764 8002 raft_consensus.cc:374] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.472802 8002 raft_consensus.cc:729] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.472929 8002 consensus_queue.cc:260] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:20.473183 8002 ts_tablet_manager.cc:1398] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:20.473459 6896 tablet_service.cc:1467] Processing CreateTablet for tablet e0e05f96dbc542eb999c16dcc0620666 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f2b695d4b3e54fdfad81f019e9ab7b4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.473681 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e05f96dbc542eb999c16dcc0620666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.473901 8000 tablet_bootstrap.cc:654] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.475116 8004 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:20.475246 8004 ts_tablet_manager.cc:1367] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 01:59:20.475699 8004 raft_consensus.cc:348] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.475813 8004 raft_consensus.cc:374] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.475847 8004 raft_consensus.cc:729] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.475872 8001 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:20.475946 8001 ts_tablet_manager.cc:1367] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:59:20.475965 8004 consensus_queue.cc:260] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.476220 8004 ts_tablet_manager.cc:1398] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:20.476410 8004 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:20.476358 8001 raft_consensus.cc:348] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:20.476459 8001 raft_consensus.cc:374] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.476492 8001 raft_consensus.cc:729] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.476603 8001 consensus_queue.cc:260] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:20.476892 8001 ts_tablet_manager.cc:1398] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:20.477092 8001 tablet_bootstrap.cc:492] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:20.477634 8004 tablet_bootstrap.cc:654] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.477960 8001 tablet_bootstrap.cc:654] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.478798 8000 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:20.478879 8000 ts_tablet_manager.cc:1367] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 01:59:20.479275 8000 raft_consensus.cc:348] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:20.479377 8000 raft_consensus.cc:374] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.479408 8000 raft_consensus.cc:729] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.479524 8000 consensus_queue.cc:260] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:20.479676 8001 tablet_bootstrap.cc:492] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:20.479749 8001 ts_tablet_manager.cc:1367] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:59:20.479794 8000 ts_tablet_manager.cc:1398] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:20.479981 8000 tablet_bootstrap.cc:492] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:20.480175 8001 raft_consensus.cc:348] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:20.480283 8001 raft_consensus.cc:374] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.480325 8001 raft_consensus.cc:729] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.480446 8001 consensus_queue.cc:260] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:20.480715 8001 ts_tablet_manager.cc:1398] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:20.480830 8000 tablet_bootstrap.cc:654] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.481820 8000 tablet_bootstrap.cc:492] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:20.482017 8000 ts_tablet_manager.cc:1367] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:59:20.482525 8000 raft_consensus.cc:348] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:20.482654 8000 raft_consensus.cc:374] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.482685 8000 raft_consensus.cc:729] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.482823 8000 consensus_queue.cc:260] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:20.483211 8000 ts_tablet_manager.cc:1398] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:20.483678 8004 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:20.483752 8004 ts_tablet_manager.cc:1367] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:59:20.484098 8004 raft_consensus.cc:348] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:20.484181 8004 raft_consensus.cc:374] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.484210 8004 raft_consensus.cc:729] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.484305 8004 consensus_queue.cc:260] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:20.484544 8004 ts_tablet_manager.cc:1398] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:20.526652 8058 raft_consensus.cc:482] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:20.526777 8058 raft_consensus.cc:504] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:20.527170 8058 leader_election.cc:288] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:20.527542 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e05f96dbc542eb999c16dcc0620666" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:20.527596 6784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e05f96dbc542eb999c16dcc0620666" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:20.527681 6917 raft_consensus.cc:2454] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:20.527737 6784 raft_consensus.cc:2454] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:20.528014 6985 leader_election.cc:302] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6101231591a458384f4ea6d1f0e0ae5, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:20.528177 8058 raft_consensus.cc:2781] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:20.528237 8058 raft_consensus.cc:482] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:20.528295 8058 raft_consensus.cc:3037] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.529215 8058 raft_consensus.cc:504] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:20.529558 8058 leader_election.cc:288] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Requested vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:20.529920 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e05f96dbc542eb999c16dcc0620666" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:20.530036 6917 raft_consensus.cc:3037] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.530074 6784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e05f96dbc542eb999c16dcc0620666" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:20.530207 6784 raft_consensus.cc:3037] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.531134 6784 raft_consensus.cc:2454] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:20.531245 6917 raft_consensus.cc:2454] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:20.531446 6984 leader_election.cc:302] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:20.531661 8058 raft_consensus.cc:2781] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:20.531759 8058 raft_consensus.cc:686] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Becoming Leader. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Running, Role: LEADER
I20240502 01:59:20.531890 8058 consensus_queue.cc:237] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:20.533589 6634 catalog_manager.cc:5484] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:20.566498 7989 raft_consensus.cc:482] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:20.566635 7989 raft_consensus.cc:504] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.567093 7989 leader_election.cc:288] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:20.567484 6784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c55b199a24432e902cce9121fc0780" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:20.567564 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c55b199a24432e902cce9121fc0780" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:20.567628 6784 raft_consensus.cc:2454] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 0.
I20240502 01:59:20.567708 7182 raft_consensus.cc:2454] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 0.
I20240502 01:59:20.567939 6851 leader_election.cc:302] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:20.568176 7989 raft_consensus.cc:2781] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:20.568228 7989 raft_consensus.cc:482] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:20.568253 7989 raft_consensus.cc:3037] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.569111 7989 raft_consensus.cc:504] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.569397 7989 leader_election.cc:288] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 election: Requested vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:20.569711 6784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c55b199a24432e902cce9121fc0780" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:20.569734 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c55b199a24432e902cce9121fc0780" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:20.569831 7182 raft_consensus.cc:3037] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.569866 6784 raft_consensus.cc:3037] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.570761 6784 raft_consensus.cc:2454] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 1.
I20240502 01:59:20.570809 7182 raft_consensus.cc:2454] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 1.
I20240502 01:59:20.571087 6851 leader_election.cc:302] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:20.571372 7989 raft_consensus.cc:2781] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:20.571581 7989 raft_consensus.cc:686] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Becoming Leader. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Running, Role: LEADER
I20240502 01:59:20.571732 7989 consensus_queue.cc:237] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.573309 6634 catalog_manager.cc:5484] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:20.610793 7989 raft_consensus.cc:482] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:20.610947 7989 raft_consensus.cc:504] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:20.611419 7989 leader_election.cc:288] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:20.611831 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818dbc9a8e054eeaa122e7919a311806" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 01:59:20.611828 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818dbc9a8e054eeaa122e7919a311806" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:20.611969 7049 raft_consensus.cc:2454] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 0.
I20240502 01:59:20.611974 7182 raft_consensus.cc:2454] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 0.
I20240502 01:59:20.612229 6852 leader_election.cc:302] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6101231591a458384f4ea6d1f0e0ae5, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:20.612386 7989 raft_consensus.cc:2781] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:20.612445 7989 raft_consensus.cc:482] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:20.612483 7989 raft_consensus.cc:3037] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.613515 7989 raft_consensus.cc:504] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:20.613881 7989 leader_election.cc:288] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 election: Requested vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:20.614161 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818dbc9a8e054eeaa122e7919a311806" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:20.614161 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818dbc9a8e054eeaa122e7919a311806" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:20.614253 7049 raft_consensus.cc:3037] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.614259 7182 raft_consensus.cc:3037] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.615032 7182 raft_consensus.cc:2454] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 1.
I20240502 01:59:20.615144 7049 raft_consensus.cc:2454] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 1.
I20240502 01:59:20.615324 6851 leader_election.cc:302] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:20.615480 7989 raft_consensus.cc:2781] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:20.615564 7989 raft_consensus.cc:686] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Becoming Leader. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Running, Role: LEADER
I20240502 01:59:20.615691 7989 consensus_queue.cc:237] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:20.617413 6634 catalog_manager.cc:5484] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
W20240502 01:59:20.727707 8072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:20.728003 8072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:20.731037 8072 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 01:59:20.824204 6784 raft_consensus.cc:1261] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:20.824697 8058 consensus_queue.cc:1035] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:20.826108 7182 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:20.826601 7989 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:20.827625 6917 raft_consensus.cc:1261] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:20.828300 7181 raft_consensus.cc:1261] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:20.829152 8058 consensus_queue.cc:1035] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:20.831986 7049 raft_consensus.cc:1261] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:20.833038 7989 consensus_queue.cc:1035] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:20.833804 7989 consensus_queue.cc:1035] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:20.842519 6784 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:20.849506 7989 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.358 ms
time per row: 0.053179 ms
I20240502 01:59:20.885931 6640 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32904:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:20.891139 6764 tablet_service.cc:1467] Processing CreateTablet for tablet 48eeb721a60c4be888041bd31cfeab7a (DEFAULT_TABLE table=rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.891472 6764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48eeb721a60c4be888041bd31cfeab7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.891505 7030 tablet_service.cc:1467] Processing CreateTablet for tablet 4154be3f1f3c428580463c419d69892e (DEFAULT_TABLE table=rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.891758 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4154be3f1f3c428580463c419d69892e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.891985 7029 tablet_service.cc:1467] Processing CreateTablet for tablet 48eeb721a60c4be888041bd31cfeab7a (DEFAULT_TABLE table=rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.892103 6896 tablet_service.cc:1467] Processing CreateTablet for tablet 39d9e48f67f34ebe850fc388ecd94ca3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.892223 7029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48eeb721a60c4be888041bd31cfeab7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.892342 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d9e48f67f34ebe850fc388ecd94ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.893853 8001 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:20.893954 8002 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:20.894389 8000 tablet_bootstrap.cc:492] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:20.894359 6765 tablet_service.cc:1467] Processing CreateTablet for tablet 4154be3f1f3c428580463c419d69892e (DEFAULT_TABLE table=rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.894601 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4154be3f1f3c428580463c419d69892e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.894815 8002 tablet_bootstrap.cc:654] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.894815 8001 tablet_bootstrap.cc:654] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.895285 8000 tablet_bootstrap.cc:654] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.896297 8000 tablet_bootstrap.cc:492] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:20.896389 8000 ts_tablet_manager.cc:1367] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:20.896766 7160 tablet_service.cc:1467] Processing CreateTablet for tablet 48eeb721a60c4be888041bd31cfeab7a (DEFAULT_TABLE table=rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.896800 8000 raft_consensus.cc:348] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.896907 8000 raft_consensus.cc:374] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.896942 8000 raft_consensus.cc:729] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.897047 7160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48eeb721a60c4be888041bd31cfeab7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.897063 8000 consensus_queue.cc:260] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.897145 8002 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:20.897212 8002 ts_tablet_manager.cc:1367] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:20.897325 8001 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:20.897328 8000 ts_tablet_manager.cc:1398] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:20.897397 8001 ts_tablet_manager.cc:1367] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:59:20.897590 8002 raft_consensus.cc:348] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:20.897679 8002 raft_consensus.cc:374] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.897704 8002 raft_consensus.cc:729] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.897809 8002 consensus_queue.cc:260] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:20.897830 8001 raft_consensus.cc:348] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.897943 8001 raft_consensus.cc:374] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.897982 8001 raft_consensus.cc:729] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.898069 8002 ts_tablet_manager.cc:1398] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:20.898089 8001 consensus_queue.cc:260] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.898218 8002 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:20.898176 6765 tablet_service.cc:1467] Processing CreateTablet for tablet 39d9e48f67f34ebe850fc388ecd94ca3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.898342 8001 ts_tablet_manager.cc:1398] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:20.898358 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d9e48f67f34ebe850fc388ecd94ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.898303 7161 tablet_service.cc:1467] Processing CreateTablet for tablet 4154be3f1f3c428580463c419d69892e (DEFAULT_TABLE table=rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.898506 8001 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:20.898584 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4154be3f1f3c428580463c419d69892e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.899058 8002 tablet_bootstrap.cc:654] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.899284 8001 tablet_bootstrap.cc:654] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.900301 7159 tablet_service.cc:1467] Processing CreateTablet for tablet 39d9e48f67f34ebe850fc388ecd94ca3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:20.900588 8004 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:20.900712 7159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d9e48f67f34ebe850fc388ecd94ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:20.901381 8004 tablet_bootstrap.cc:654] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.902179 8001 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:20.902246 8001 ts_tablet_manager.cc:1367] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:59:20.902709 8001 raft_consensus.cc:348] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:20.902817 8001 raft_consensus.cc:374] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.902853 8001 raft_consensus.cc:729] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.902962 8001 consensus_queue.cc:260] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:20.903265 8001 ts_tablet_manager.cc:1398] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:20.903985 8004 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:20.904076 8004 ts_tablet_manager.cc:1367] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:59:20.904516 8004 raft_consensus.cc:348] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.904626 8004 raft_consensus.cc:374] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.904654 8004 raft_consensus.cc:729] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.904758 8004 consensus_queue.cc:260] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.905025 8004 ts_tablet_manager.cc:1398] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:20.905169 8004 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:20.905939 8004 tablet_bootstrap.cc:654] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.906358 8002 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:20.906579 8002 ts_tablet_manager.cc:1367] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240502 01:59:20.906975 8004 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:20.907073 8004 ts_tablet_manager.cc:1367] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:20.907518 8004 raft_consensus.cc:348] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:20.907619 8004 raft_consensus.cc:374] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.907661 8004 raft_consensus.cc:729] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.907030 8002 raft_consensus.cc:348] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.907730 8002 raft_consensus.cc:374] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.907771 8002 raft_consensus.cc:729] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.907778 8004 consensus_queue.cc:260] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:20.907907 8002 consensus_queue.cc:260] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.908155 8004 ts_tablet_manager.cc:1398] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:20.908164 8002 ts_tablet_manager.cc:1398] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:59:20.908295 8004 tablet_bootstrap.cc:492] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:20.908331 8002 tablet_bootstrap.cc:492] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:20.909045 8004 tablet_bootstrap.cc:654] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.909904 8002 tablet_bootstrap.cc:654] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:20.910956 8002 tablet_bootstrap.cc:492] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:20.911037 8002 ts_tablet_manager.cc:1367] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:59:20.911419 8002 raft_consensus.cc:348] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.911499 8002 raft_consensus.cc:374] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.911520 8002 raft_consensus.cc:729] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.911613 8002 consensus_queue.cc:260] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.911846 8002 ts_tablet_manager.cc:1398] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:20.912204 8004 tablet_bootstrap.cc:492] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:20.912278 8004 ts_tablet_manager.cc:1367] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 01:59:20.912591 8004 raft_consensus.cc:348] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.912662 8004 raft_consensus.cc:374] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:20.912691 8004 raft_consensus.cc:729] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:20.912820 8004 consensus_queue.cc:260] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.913076 8004 ts_tablet_manager.cc:1398] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:20.935714 8069 raft_consensus.cc:482] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:20.935837 8069 raft_consensus.cc:504] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.936190 8069 leader_election.cc:288] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:20.936538 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d9e48f67f34ebe850fc388ecd94ca3" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:20.936576 7181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d9e48f67f34ebe850fc388ecd94ca3" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:20.936693 6785 raft_consensus.cc:2454] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 0.
I20240502 01:59:20.936692 7181 raft_consensus.cc:2454] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 0.
I20240502 01:59:20.936964 6851 leader_election.cc:302] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:20.937110 8069 raft_consensus.cc:2781] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:20.937170 8069 raft_consensus.cc:482] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:20.937217 8069 raft_consensus.cc:3037] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.938016 8069 raft_consensus.cc:504] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.938351 8069 leader_election.cc:288] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 election: Requested vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:20.938657 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d9e48f67f34ebe850fc388ecd94ca3" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:20.938719 7181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d9e48f67f34ebe850fc388ecd94ca3" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:20.938763 6785 raft_consensus.cc:3037] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.938870 7181 raft_consensus.cc:3037] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:20.939546 6785 raft_consensus.cc:2454] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 1.
I20240502 01:59:20.939708 7181 raft_consensus.cc:2454] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 1.
I20240502 01:59:20.939855 6851 leader_election.cc:302] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:20.940038 8069 raft_consensus.cc:2781] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:20.940138 8069 raft_consensus.cc:686] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Becoming Leader. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Running, Role: LEADER
I20240502 01:59:20.940263 8069 consensus_queue.cc:237] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:20.941963 6636 catalog_manager.cc:5484] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:21.020864 8057 raft_consensus.cc:482] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:21.021014 8057 raft_consensus.cc:504] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.021458 8057 leader_election.cc:288] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:21.021824 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4154be3f1f3c428580463c419d69892e" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 01:59:21.021904 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4154be3f1f3c428580463c419d69892e" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:21.022001 7049 raft_consensus.cc:2454] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 0.
I20240502 01:59:21.022089 6785 raft_consensus.cc:2454] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 0.
I20240502 01:59:21.022363 7117 leader_election.cc:302] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:21.022547 8057 raft_consensus.cc:2781] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:21.022627 8057 raft_consensus.cc:482] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:21.022693 8057 raft_consensus.cc:3037] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.023604 8057 raft_consensus.cc:504] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.023937 8057 leader_election.cc:288] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 election: Requested vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:21.024215 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4154be3f1f3c428580463c419d69892e" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:21.024317 7049 raft_consensus.cc:3037] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.024425 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4154be3f1f3c428580463c419d69892e" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:21.024526 6785 raft_consensus.cc:3037] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.025291 7049 raft_consensus.cc:2454] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 1.
I20240502 01:59:21.025480 6785 raft_consensus.cc:2454] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 1.
I20240502 01:59:21.025635 7117 leader_election.cc:302] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:21.025781 8057 raft_consensus.cc:2781] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:21.025887 8057 raft_consensus.cc:686] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Becoming Leader. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Running, Role: LEADER
I20240502 01:59:21.026027 8057 consensus_queue.cc:237] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.027758 6636 catalog_manager.cc:5484] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 86bc8001c3464b929cd5c873861baa0c (127.27.111.196). New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:21.060873 8058 raft_consensus.cc:482] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:21.060987 8058 raft_consensus.cc:504] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.061331 8058 leader_election.cc:288] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:21.061668 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48eeb721a60c4be888041bd31cfeab7a" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:21.061787 6785 raft_consensus.cc:2454] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:21.061745 7181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48eeb721a60c4be888041bd31cfeab7a" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:21.061913 7181 raft_consensus.cc:2454] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:21.062093 6984 leader_election.cc:302] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:21.062247 8058 raft_consensus.cc:2781] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:21.062294 8058 raft_consensus.cc:482] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:21.062320 8058 raft_consensus.cc:3037] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.065624 8058 raft_consensus.cc:504] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.066015 8058 leader_election.cc:288] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Requested vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:21.066465 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48eeb721a60c4be888041bd31cfeab7a" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:21.066473 7181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48eeb721a60c4be888041bd31cfeab7a" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:21.066593 7181 raft_consensus.cc:3037] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.066584 6785 raft_consensus.cc:3037] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.067440 6785 raft_consensus.cc:2454] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:21.067714 6984 leader_election.cc:302] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:21.067864 8058 raft_consensus.cc:2781] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:21.067968 8058 raft_consensus.cc:686] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Becoming Leader. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Running, Role: LEADER
I20240502 01:59:21.068113 8058 consensus_queue.cc:237] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.069799 6636 catalog_manager.cc:5484] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 01:59:21.087177 7181 raft_consensus.cc:2454] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
W20240502 01:59:21.197853 8120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:21.198091 8120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:21.201141 8120 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 01:59:21.290190 7181 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:21.290697 8058 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:21.291478 6785 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:21.292582 6785 raft_consensus.cc:1261] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:21.293140 7836 consensus_queue.cc:1035] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:21.296315 7947 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:21.297991 6784 raft_consensus.cc:1261] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:21.298293 7180 raft_consensus.cc:1261] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:21.299209 7049 raft_consensus.cc:1261] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:21.299711 8106 consensus_queue.cc:1035] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:21.301477 7836 consensus_queue.cc:1035] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:21.301496 8106 consensus_queue.cc:1035] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:21.306246 8052 mvcc.cc:205] Tried to move back new op lower bound from 7023063700662345728 to 7023063699736363008. Current Snapshot: MvccSnapshot[applied={T|T < 7023063700641210368}]
I20240502 01:59:21.326790 7944 mvcc.cc:205] Tried to move back new op lower bound from 7023063700682719232 to 7023063699564081152. Current Snapshot: MvccSnapshot[applied={T|T < 7023063700644679680}]
I20240502 01:59:21.331934 7972 mvcc.cc:205] Tried to move back new op lower bound from 7023063700644679680 to 7023063699564081152. Current Snapshot: MvccSnapshot[applied={T|T < 7023063700644679680}]
INSERT report
rows total: 2000
time total: 107.491 ms
time per row: 0.0537456 ms
I20240502 01:59:21.352994 6641 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32904:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:21.359822 6896 tablet_service.cc:1467] Processing CreateTablet for tablet 034cf1c3d47a4f7785c16d16815c229d (DEFAULT_TABLE table=rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.360092 6765 tablet_service.cc:1467] Processing CreateTablet for tablet 034cf1c3d47a4f7785c16d16815c229d (DEFAULT_TABLE table=rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.360159 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034cf1c3d47a4f7785c16d16815c229d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.360388 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034cf1c3d47a4f7785c16d16815c229d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.361161 6764 tablet_service.cc:1467] Processing CreateTablet for tablet 9313079c48c94ab1b7e022a91befafcd (DEFAULT_TABLE table=rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.361393 6764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9313079c48c94ab1b7e022a91befafcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.361443 7029 tablet_service.cc:1467] Processing CreateTablet for tablet 034cf1c3d47a4f7785c16d16815c229d (DEFAULT_TABLE table=rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.361719 7029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034cf1c3d47a4f7785c16d16815c229d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.362802 8000 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:21.363312 6896 tablet_service.cc:1467] Processing CreateTablet for tablet ed7b67af7b1c4255a7c2377570b58e70 (DEFAULT_TABLE table=rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.363559 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed7b67af7b1c4255a7c2377570b58e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.363775 8000 tablet_bootstrap.cc:654] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.363807 8001 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:21.364010 7030 tablet_service.cc:1467] Processing CreateTablet for tablet ed7b67af7b1c4255a7c2377570b58e70 (DEFAULT_TABLE table=rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.364224 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed7b67af7b1c4255a7c2377570b58e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.364302 8002 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:21.364675 8001 tablet_bootstrap.cc:654] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.365162 8002 tablet_bootstrap.cc:654] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.365942 6894 tablet_service.cc:1467] Processing CreateTablet for tablet 9313079c48c94ab1b7e022a91befafcd (DEFAULT_TABLE table=rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.366171 6894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9313079c48c94ab1b7e022a91befafcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.366271 8002 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:21.366289 7028 tablet_service.cc:1467] Processing CreateTablet for tablet 9313079c48c94ab1b7e022a91befafcd (DEFAULT_TABLE table=rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.366375 8002 ts_tablet_manager.cc:1367] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:21.366549 7028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9313079c48c94ab1b7e022a91befafcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.366935 8002 raft_consensus.cc:348] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.367053 8002 raft_consensus.cc:374] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.367094 8002 raft_consensus.cc:729] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.367208 8002 consensus_queue.cc:260] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.367493 8002 ts_tablet_manager.cc:1398] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:21.367671 8002 tablet_bootstrap.cc:492] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:21.368670 8002 tablet_bootstrap.cc:654] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.368744 8001 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:21.368822 8001 ts_tablet_manager.cc:1367] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:21.369349 8001 raft_consensus.cc:348] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.369460 8001 raft_consensus.cc:374] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.369516 8001 raft_consensus.cc:729] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.369443 7159 tablet_service.cc:1467] Processing CreateTablet for tablet ed7b67af7b1c4255a7c2377570b58e70 (DEFAULT_TABLE table=rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.369621 8001 consensus_queue.cc:260] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.369774 7159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed7b67af7b1c4255a7c2377570b58e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.369880 8001 ts_tablet_manager.cc:1398] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:21.369930 8002 tablet_bootstrap.cc:492] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:21.370008 8001 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:21.369998 8002 ts_tablet_manager.cc:1367] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:59:21.370529 8002 raft_consensus.cc:348] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.370625 8002 raft_consensus.cc:374] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.370654 8002 raft_consensus.cc:729] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.370774 8001 tablet_bootstrap.cc:654] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.370772 8002 consensus_queue.cc:260] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.371042 8002 ts_tablet_manager.cc:1398] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:21.371793 8000 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:21.371873 8000 ts_tablet_manager.cc:1367] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:59:21.372197 8000 raft_consensus.cc:348] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.372280 8000 raft_consensus.cc:374] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.372313 8000 raft_consensus.cc:729] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.372442 8000 consensus_queue.cc:260] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.372534 8004 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:21.372694 8000 ts_tablet_manager.cc:1398] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:21.373088 8000 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:21.373327 8004 tablet_bootstrap.cc:654] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.373811 8000 tablet_bootstrap.cc:654] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.374536 8004 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:21.374629 8004 ts_tablet_manager.cc:1367] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:59:21.375099 8004 raft_consensus.cc:348] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.375202 8004 raft_consensus.cc:374] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.375236 8004 raft_consensus.cc:729] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.375362 8004 consensus_queue.cc:260] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.375681 8004 ts_tablet_manager.cc:1398] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:21.375771 8001 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:21.375857 8001 ts_tablet_manager.cc:1367] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:59:21.375928 8000 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:21.376005 8000 ts_tablet_manager.cc:1367] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:59:21.376199 8001 raft_consensus.cc:348] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.376283 8001 raft_consensus.cc:374] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.376317 8001 raft_consensus.cc:729] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.376327 8000 raft_consensus.cc:348] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.376421 8000 raft_consensus.cc:374] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.376456 8000 raft_consensus.cc:729] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.376439 8001 consensus_queue.cc:260] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.376685 8001 ts_tablet_manager.cc:1398] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:21.376833 8001 tablet_bootstrap.cc:492] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:21.377415 8000 consensus_queue.cc:260] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.377632 8001 tablet_bootstrap.cc:654] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.377676 8000 ts_tablet_manager.cc:1398] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:59:21.377789 8000 tablet_bootstrap.cc:492] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:21.378490 8000 tablet_bootstrap.cc:654] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.379665 8001 tablet_bootstrap.cc:492] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:21.379731 8001 ts_tablet_manager.cc:1367] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:21.380137 8001 raft_consensus.cc:348] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.380208 8001 raft_consensus.cc:374] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.380227 8001 raft_consensus.cc:729] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.380311 8001 consensus_queue.cc:260] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.380497 8001 ts_tablet_manager.cc:1398] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:21.381086 8000 tablet_bootstrap.cc:492] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:21.381171 8000 ts_tablet_manager.cc:1367] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:59:21.381556 8000 raft_consensus.cc:348] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.381686 8000 raft_consensus.cc:374] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.381727 8000 raft_consensus.cc:729] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.381848 8000 consensus_queue.cc:260] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.382128 8000 ts_tablet_manager.cc:1398] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:21.443920 8154 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:21.444036 8154 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.444449 8154 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:21.444839 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034cf1c3d47a4f7785c16d16815c229d" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:21.444895 6783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034cf1c3d47a4f7785c16d16815c229d" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:21.444965 6917 raft_consensus.cc:2454] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:21.445077 6783 raft_consensus.cc:2454] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:21.445240 6985 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6101231591a458384f4ea6d1f0e0ae5, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:21.445427 8154 raft_consensus.cc:2781] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:21.445483 8154 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:21.445533 8154 raft_consensus.cc:3037] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.446439 8154 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.446817 8154 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Requested vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:21.447088 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034cf1c3d47a4f7785c16d16815c229d" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:21.447098 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034cf1c3d47a4f7785c16d16815c229d" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:21.447197 6917 raft_consensus.cc:3037] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.447206 6785 raft_consensus.cc:3037] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.447966 6785 raft_consensus.cc:2454] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:21.448053 6917 raft_consensus.cc:2454] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:21.448231 6984 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:21.448439 8154 raft_consensus.cc:2781] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:21.448534 8154 raft_consensus.cc:686] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Becoming Leader. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Running, Role: LEADER
I20240502 01:59:21.448680 8154 consensus_queue.cc:237] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.450199 6641 catalog_manager.cc:5484] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 01:59:21.528439 8154 raft_consensus.cc:482] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:21.528573 8154 raft_consensus.cc:504] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.528949 8154 leader_election.cc:288] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:21.529320 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9313079c48c94ab1b7e022a91befafcd" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:21.529357 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9313079c48c94ab1b7e022a91befafcd" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:21.529440 6917 raft_consensus.cc:2454] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:21.529539 6785 raft_consensus.cc:2454] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:21.529734 6985 leader_election.cc:302] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6101231591a458384f4ea6d1f0e0ae5, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:21.529911 8154 raft_consensus.cc:2781] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:21.529975 8154 raft_consensus.cc:482] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:21.530011 8154 raft_consensus.cc:3037] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.530882 8154 raft_consensus.cc:504] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.531198 8154 leader_election.cc:288] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Requested vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:21.531466 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9313079c48c94ab1b7e022a91befafcd" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:21.531595 6917 raft_consensus.cc:3037] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.531572 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9313079c48c94ab1b7e022a91befafcd" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:21.531684 6785 raft_consensus.cc:3037] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.532609 6917 raft_consensus.cc:2454] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:21.532613 6785 raft_consensus.cc:2454] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:21.532927 6985 leader_election.cc:302] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6101231591a458384f4ea6d1f0e0ae5, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:21.533092 8154 raft_consensus.cc:2781] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:21.533180 8154 raft_consensus.cc:686] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Becoming Leader. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Running, Role: LEADER
I20240502 01:59:21.533321 8154 consensus_queue.cc:237] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.534934 6641 catalog_manager.cc:5484] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 01:59:21.583709 8069 raft_consensus.cc:482] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:21.583858 8069 raft_consensus.cc:504] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.584270 8069 leader_election.cc:288] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:21.584633 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7b67af7b1c4255a7c2377570b58e70" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 01:59:21.584632 7181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7b67af7b1c4255a7c2377570b58e70" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:21.584776 7049 raft_consensus.cc:2454] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 0.
I20240502 01:59:21.584776 7181 raft_consensus.cc:2454] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 0.
I20240502 01:59:21.585052 6851 leader_election.cc:302] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:21.585211 8069 raft_consensus.cc:2781] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:21.585268 8069 raft_consensus.cc:482] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:21.585294 8069 raft_consensus.cc:3037] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.586128 8069 raft_consensus.cc:504] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.586478 8069 leader_election.cc:288] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 election: Requested vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:21.586763 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7b67af7b1c4255a7c2377570b58e70" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:21.586794 7181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7b67af7b1c4255a7c2377570b58e70" candidate_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:21.586897 7181 raft_consensus.cc:3037] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.586869 7049 raft_consensus.cc:3037] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.587610 7049 raft_consensus.cc:2454] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 1.
I20240502 01:59:21.587646 7181 raft_consensus.cc:2454] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7bcff2fef614e7a9981554c9e6db3c8 in term 1.
I20240502 01:59:21.587903 6851 leader_election.cc:302] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:21.588076 8069 raft_consensus.cc:2781] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:21.588160 8069 raft_consensus.cc:686] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Becoming Leader. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Running, Role: LEADER
I20240502 01:59:21.588302 8069 consensus_queue.cc:237] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.589951 6641 catalog_manager.cc:5484] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
W20240502 01:59:21.715463 8173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:21.715700 8173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:21.718676 8173 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 01:59:21.794695 6916 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:21.794780 6917 raft_consensus.cc:1261] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:21.795370 7792 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:21.795423 6785 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:21.795665 7792 consensus_queue.cc:1035] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:21.795728 6783 raft_consensus.cc:1261] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:21.796164 7947 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:21.796411 7947 consensus_queue.cc:1035] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:21.804814 7181 raft_consensus.cc:1261] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:21.807126 8106 consensus_queue.cc:1035] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:21.808066 7049 raft_consensus.cc:1261] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:21.809633 8106 consensus_queue.cc:1035] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:21.824303 7943 mvcc.cc:205] Tried to move back new op lower bound from 7023063702703755264 to 7023063701294858240. Current Snapshot: MvccSnapshot[applied={T|T < 7023063702703755264}]
INSERT report
rows total: 2000
time total: 94.8062 ms
time per row: 0.0474031 ms
I20240502 01:59:21.856683 6643 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32904:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:21.862459 6764 tablet_service.cc:1467] Processing CreateTablet for tablet a4fb9af11abc49d496b5c3d8048c22c8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.862797 6764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4fb9af11abc49d496b5c3d8048c22c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.862747 6894 tablet_service.cc:1467] Processing CreateTablet for tablet a4fb9af11abc49d496b5c3d8048c22c8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.863037 6894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4fb9af11abc49d496b5c3d8048c22c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.863353 6765 tablet_service.cc:1467] Processing CreateTablet for tablet 7345f51bd6f84b3880838c2460bafb75 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.863546 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7345f51bd6f84b3880838c2460bafb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.863530 7160 tablet_service.cc:1467] Processing CreateTablet for tablet 7fa358140eb742b587a4d6a203738b4f (DEFAULT_TABLE table=rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.863775 7160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fa358140eb742b587a4d6a203738b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.865029 8002 tablet_bootstrap.cc:492] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:21.865239 6896 tablet_service.cc:1467] Processing CreateTablet for tablet 7fa358140eb742b587a4d6a203738b4f (DEFAULT_TABLE table=rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.865443 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fa358140eb742b587a4d6a203738b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.865551 7159 tablet_service.cc:1467] Processing CreateTablet for tablet a4fb9af11abc49d496b5c3d8048c22c8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.865792 7159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4fb9af11abc49d496b5c3d8048c22c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.865973 8002 tablet_bootstrap.cc:654] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.865973 7161 tablet_service.cc:1467] Processing CreateTablet for tablet 7345f51bd6f84b3880838c2460bafb75 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.866169 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7345f51bd6f84b3880838c2460bafb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.867300 8000 tablet_bootstrap.cc:492] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:21.867942 8002 tablet_bootstrap.cc:492] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:21.867950 8004 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:21.868014 8002 ts_tablet_manager.cc:1367] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:21.868074 8000 tablet_bootstrap.cc:654] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.868345 8002 raft_consensus.cc:348] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:21.868446 8002 raft_consensus.cc:374] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.868476 8002 raft_consensus.cc:729] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.868614 8002 consensus_queue.cc:260] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:21.868840 8004 tablet_bootstrap.cc:654] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.868882 8002 ts_tablet_manager.cc:1398] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:21.869014 8002 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:21.869766 7028 tablet_service.cc:1467] Processing CreateTablet for tablet 7fa358140eb742b587a4d6a203738b4f (DEFAULT_TABLE table=rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.870034 8000 tablet_bootstrap.cc:492] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:21.870045 7028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fa358140eb742b587a4d6a203738b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.870085 8002 tablet_bootstrap.cc:654] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.870105 8000 ts_tablet_manager.cc:1367] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:59:21.870611 8000 raft_consensus.cc:348] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:21.870700 8000 raft_consensus.cc:374] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.870728 8000 raft_consensus.cc:729] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.870824 8000 consensus_queue.cc:260] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:21.871078 8000 ts_tablet_manager.cc:1398] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:21.871088 8002 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:21.871163 8002 ts_tablet_manager.cc:1367] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:59:21.871246 8000 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:21.871776 8002 raft_consensus.cc:348] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.871891 8002 raft_consensus.cc:374] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.871927 8002 raft_consensus.cc:729] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.872092 8000 tablet_bootstrap.cc:654] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.872056 8002 consensus_queue.cc:260] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.872359 8002 ts_tablet_manager.cc:1398] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:21.872334 7030 tablet_service.cc:1467] Processing CreateTablet for tablet 7345f51bd6f84b3880838c2460bafb75 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:21.872565 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7345f51bd6f84b3880838c2460bafb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:21.873091 8000 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:21.873375 8000 ts_tablet_manager.cc:1367] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:21.873859 8000 raft_consensus.cc:348] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.873998 8000 raft_consensus.cc:374] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.874040 8000 raft_consensus.cc:729] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.874222 8000 consensus_queue.cc:260] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.874414 8001 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:21.874560 8000 ts_tablet_manager.cc:1398] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:21.875295 8001 tablet_bootstrap.cc:654] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.876363 8001 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:21.876442 8001 ts_tablet_manager.cc:1367] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:21.876812 8004 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:21.876894 8004 ts_tablet_manager.cc:1367] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 01:59:21.876854 8001 raft_consensus.cc:348] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.876960 8001 raft_consensus.cc:374] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.876992 8001 raft_consensus.cc:729] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.877097 8001 consensus_queue.cc:260] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.877358 8001 ts_tablet_manager.cc:1398] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:21.877311 8004 raft_consensus.cc:348] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.877415 8004 raft_consensus.cc:374] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.877498 8004 raft_consensus.cc:729] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.877511 8001 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:21.877621 8004 consensus_queue.cc:260] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.877880 8004 ts_tablet_manager.cc:1398] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:21.878039 8004 tablet_bootstrap.cc:492] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:21.878298 8001 tablet_bootstrap.cc:654] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.878728 8004 tablet_bootstrap.cc:654] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.879388 8001 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:21.879514 8001 ts_tablet_manager.cc:1367] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:59:21.879940 8001 raft_consensus.cc:348] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.880035 8001 raft_consensus.cc:374] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.880065 8001 raft_consensus.cc:729] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.880167 8001 consensus_queue.cc:260] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.880425 8001 ts_tablet_manager.cc:1398] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:21.883057 8004 tablet_bootstrap.cc:492] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:21.883134 8004 ts_tablet_manager.cc:1367] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:21.883509 8004 raft_consensus.cc:348] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:21.883585 8004 raft_consensus.cc:374] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.883616 8004 raft_consensus.cc:729] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.883740 8004 consensus_queue.cc:260] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:21.884003 8004 ts_tablet_manager.cc:1398] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:21.884352 8004 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:21.885763 8004 tablet_bootstrap.cc:654] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:21.886791 8004 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:21.886873 8004 ts_tablet_manager.cc:1367] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:21.887166 8004 raft_consensus.cc:348] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.887239 8004 raft_consensus.cc:374] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:21.887272 8004 raft_consensus.cc:729] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:21.887355 8004 consensus_queue.cc:260] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:21.887562 8004 ts_tablet_manager.cc:1398] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:21.889787 8160 raft_consensus.cc:482] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:21.889884 8160 raft_consensus.cc:504] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.890198 8160 leader_election.cc:288] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:21.890565 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa358140eb742b587a4d6a203738b4f" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:21.890609 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa358140eb742b587a4d6a203738b4f" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 01:59:21.890707 7049 raft_consensus.cc:2454] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 0.
I20240502 01:59:21.890707 6916 raft_consensus.cc:2454] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 0.
I20240502 01:59:21.890944 7117 leader_election.cc:302] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:21.891083 8160 raft_consensus.cc:2781] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:21.891140 8160 raft_consensus.cc:482] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:21.891188 8160 raft_consensus.cc:3037] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.891875 8160 raft_consensus.cc:504] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.892139 8160 leader_election.cc:288] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 election: Requested vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:21.892421 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa358140eb742b587a4d6a203738b4f" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:21.892463 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fa358140eb742b587a4d6a203738b4f" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:21.892535 6916 raft_consensus.cc:3037] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.892544 7049 raft_consensus.cc:3037] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.893433 6916 raft_consensus.cc:2454] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 1.
I20240502 01:59:21.893433 7049 raft_consensus.cc:2454] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 1.
I20240502 01:59:21.893690 7117 leader_election.cc:302] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:21.893860 8160 raft_consensus.cc:2781] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:21.893952 8160 raft_consensus.cc:686] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Becoming Leader. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Running, Role: LEADER
I20240502 01:59:21.894083 8160 consensus_queue.cc:237] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:21.895603 6641 catalog_manager.cc:5484] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 86bc8001c3464b929cd5c873861baa0c (127.27.111.196). New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:21.953603 7832 raft_consensus.cc:482] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:21.953723 7832 raft_consensus.cc:504] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:21.954144 7832 leader_election.cc:288] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:21.954329 8160 raft_consensus.cc:482] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:21.954453 8160 raft_consensus.cc:504] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:21.954555 7181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4fb9af11abc49d496b5c3d8048c22c8" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:21.954550 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4fb9af11abc49d496b5c3d8048c22c8" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:21.954653 7181 raft_consensus.cc:2454] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:21.954695 6916 raft_consensus.cc:2454] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:21.954973 8160 leader_election.cc:288] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:21.954967 6721 leader_election.cc:302] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:21.955206 7832 raft_consensus.cc:2781] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:21.955186 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4fb9af11abc49d496b5c3d8048c22c8" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:21.955322 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4fb9af11abc49d496b5c3d8048c22c8" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:21.955371 7832 raft_consensus.cc:482] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:21.955417 7832 raft_consensus.cc:3037] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.955420 6916 raft_consensus.cc:2454] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 0.
I20240502 01:59:21.955677 7117 leader_election.cc:302] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:21.955857 8160 raft_consensus.cc:2781] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:21.955916 8160 raft_consensus.cc:482] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:21.955942 8160 raft_consensus.cc:3037] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.956354 7832 raft_consensus.cc:504] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:21.956581 6785 raft_consensus.cc:2379] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 86bc8001c3464b929cd5c873861baa0c in current term 1: Already voted for candidate 85e78945910a4e1abfa101d2edc3644e in this term.
I20240502 01:59:21.956737 7832 leader_election.cc:288] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Requested vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:21.957070 7181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4fb9af11abc49d496b5c3d8048c22c8" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:21.957115 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4fb9af11abc49d496b5c3d8048c22c8" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:21.957104 8160 raft_consensus.cc:504] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:21.957207 6916 raft_consensus.cc:3037] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:21.957334 7181 raft_consensus.cc:2379] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 85e78945910a4e1abfa101d2edc3644e in current term 1: Already voted for candidate 86bc8001c3464b929cd5c873861baa0c in this term.
I20240502 01:59:21.957490 8160 leader_election.cc:288] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 election: Requested vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:21.957770 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4fb9af11abc49d496b5c3d8048c22c8" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:21.957803 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4fb9af11abc49d496b5c3d8048c22c8" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:21.957912 6785 raft_consensus.cc:2379] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 86bc8001c3464b929cd5c873861baa0c in current term 1: Already voted for candidate 85e78945910a4e1abfa101d2edc3644e in this term.
I20240502 01:59:21.958194 7116 leader_election.cc:302] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 85e78945910a4e1abfa101d2edc3644e, e7bcff2fef614e7a9981554c9e6db3c8
I20240502 01:59:21.958256 6916 raft_consensus.cc:2454] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 1.
I20240502 01:59:21.958369 8160 raft_consensus.cc:2726] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:59:21.958525 6721 leader_election.cc:302] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, e7bcff2fef614e7a9981554c9e6db3c8; no voters: 86bc8001c3464b929cd5c873861baa0c
I20240502 01:59:21.958755 7832 raft_consensus.cc:2781] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:21.958846 7832 raft_consensus.cc:686] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Becoming Leader. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:21.958992 7832 consensus_queue.cc:237] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:21.960803 6641 catalog_manager.cc:5484] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: term changed from 0 to 1, leader changed from <none> to 85e78945910a4e1abfa101d2edc3644e (127.27.111.193). New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 01:59:22.057652 8058 raft_consensus.cc:482] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:22.057765 8058 raft_consensus.cc:504] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:22.058180 8058 leader_election.cc:288] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:22.058563 7181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7345f51bd6f84b3880838c2460bafb75" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:22.058661 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7345f51bd6f84b3880838c2460bafb75" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:22.058738 7181 raft_consensus.cc:2454] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:22.058794 6785 raft_consensus.cc:2454] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:22.059058 6984 leader_election.cc:302] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:22.059206 8058 raft_consensus.cc:2781] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:22.059262 8058 raft_consensus.cc:482] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:22.059306 8058 raft_consensus.cc:3037] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:22.060178 8058 raft_consensus.cc:504] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:22.060460 8058 leader_election.cc:288] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Requested vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:22.060791 7181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7345f51bd6f84b3880838c2460bafb75" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:22.060869 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7345f51bd6f84b3880838c2460bafb75" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:22.060905 7181 raft_consensus.cc:3037] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:22.060986 6785 raft_consensus.cc:3037] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:22.061918 6785 raft_consensus.cc:2454] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:22.061918 7181 raft_consensus.cc:2454] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:22.062211 6984 leader_election.cc:302] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:22.062436 8058 raft_consensus.cc:2781] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:22.062544 8058 raft_consensus.cc:686] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Becoming Leader. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Running, Role: LEADER
I20240502 01:59:22.062683 8058 consensus_queue.cc:237] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:22.064293 6641 catalog_manager.cc:5484] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
W20240502 01:59:22.214417 8223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:22.214648 8223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:22.217625 8223 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 01:59:22.296120 8160 consensus_queue.cc:1035] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:22.305450 6917 raft_consensus.cc:1261] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:22.305761 7181 raft_consensus.cc:1261] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:22.306283 7835 consensus_queue.cc:1035] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:22.306624 7835 consensus_queue.cc:1035] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:22.310096 7049 raft_consensus.cc:1261] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:22.310798 6785 raft_consensus.cc:1261] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:22.311077 7836 consensus_queue.cc:1035] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:22.311487 8058 consensus_queue.cc:1035] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:22.311077 7180 raft_consensus.cc:1261] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:22.313733 8058 consensus_queue.cc:1035] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 120.48 ms
time per row: 0.0602398 ms
I20240502 01:59:22.380864 6638 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32904:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:22.386122 7161 tablet_service.cc:1467] Processing CreateTablet for tablet 1c54b912f802461c8cac092ac3222c87 (DEFAULT_TABLE table=rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:22.386122 6765 tablet_service.cc:1467] Processing CreateTablet for tablet 1c54b912f802461c8cac092ac3222c87 (DEFAULT_TABLE table=rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:22.386335 6896 tablet_service.cc:1467] Processing CreateTablet for tablet 1c54b912f802461c8cac092ac3222c87 (DEFAULT_TABLE table=rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:22.386468 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c54b912f802461c8cac092ac3222c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:22.386468 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c54b912f802461c8cac092ac3222c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:22.386683 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c54b912f802461c8cac092ac3222c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:22.387095 7030 tablet_service.cc:1467] Processing CreateTablet for tablet a0548274c47148d9b611fd3c978a0c34 (DEFAULT_TABLE table=rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:22.387363 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0548274c47148d9b611fd3c978a0c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:22.388720 8004 tablet_bootstrap.cc:492] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:22.389219 7161 tablet_service.cc:1467] Processing CreateTablet for tablet 14e3c78c65334b52aca79d67c4bf45a2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:22.389437 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e3c78c65334b52aca79d67c4bf45a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:22.389571 8004 tablet_bootstrap.cc:654] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:22.389793 6894 tablet_service.cc:1467] Processing CreateTablet for tablet 14e3c78c65334b52aca79d67c4bf45a2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:22.390036 6894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e3c78c65334b52aca79d67c4bf45a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:22.390194 7030 tablet_service.cc:1467] Processing CreateTablet for tablet 14e3c78c65334b52aca79d67c4bf45a2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:22.390475 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e3c78c65334b52aca79d67c4bf45a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:22.390955 6765 tablet_service.cc:1467] Processing CreateTablet for tablet a0548274c47148d9b611fd3c978a0c34 (DEFAULT_TABLE table=rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:22.391201 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0548274c47148d9b611fd3c978a0c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:22.391753 8271 tablet_bootstrap.cc:492] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:22.392607 6896 tablet_service.cc:1467] Processing CreateTablet for tablet a0548274c47148d9b611fd3c978a0c34 (DEFAULT_TABLE table=rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:22.392688 8271 tablet_bootstrap.cc:654] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:22.392853 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0548274c47148d9b611fd3c978a0c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:22.392975 8272 tablet_bootstrap.cc:492] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:22.393847 8271 tablet_bootstrap.cc:492] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:22.393954 8271 ts_tablet_manager.cc:1367] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:22.393992 8272 tablet_bootstrap.cc:654] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:22.394483 8271 raft_consensus.cc:348] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.394595 8271 raft_consensus.cc:374] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:22.394631 8271 raft_consensus.cc:729] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:22.394776 8271 consensus_queue.cc:260] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.395087 8271 ts_tablet_manager.cc:1398] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:22.395093 8273 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:22.395188 8004 tablet_bootstrap.cc:492] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:22.395260 8271 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:22.395269 8004 ts_tablet_manager.cc:1367] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:59:22.395702 8004 raft_consensus.cc:348] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.395808 8004 raft_consensus.cc:374] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:22.395853 8004 raft_consensus.cc:729] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:22.395917 8273 tablet_bootstrap.cc:654] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:22.395977 8004 consensus_queue.cc:260] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.396260 8271 tablet_bootstrap.cc:654] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:22.396262 8004 ts_tablet_manager.cc:1398] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:22.396437 8004 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:22.397190 8273 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:22.397234 8004 tablet_bootstrap.cc:654] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:22.397277 8273 ts_tablet_manager.cc:1367] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:22.397687 8273 raft_consensus.cc:348] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.397778 8273 raft_consensus.cc:374] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:22.397811 8273 raft_consensus.cc:729] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:22.397959 8273 consensus_queue.cc:260] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.398259 8273 ts_tablet_manager.cc:1398] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:22.398439 8273 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:22.399039 8272 tablet_bootstrap.cc:492] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:22.399137 8272 ts_tablet_manager.cc:1367] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:59:22.399300 8273 tablet_bootstrap.cc:654] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:22.399565 8272 raft_consensus.cc:348] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.399662 8272 raft_consensus.cc:374] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:22.399691 8272 raft_consensus.cc:729] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:22.399804 8272 consensus_queue.cc:260] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.400089 8272 ts_tablet_manager.cc:1398] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:22.400246 8272 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:22.401130 8272 tablet_bootstrap.cc:654] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:22.401232 8004 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:22.401311 8004 ts_tablet_manager.cc:1367] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:59:22.401515 8271 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:22.401590 8271 ts_tablet_manager.cc:1367] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:59:22.401739 8004 raft_consensus.cc:348] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:22.401839 8004 raft_consensus.cc:374] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:22.401883 8004 raft_consensus.cc:729] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:22.401995 8004 consensus_queue.cc:260] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:22.402004 8271 raft_consensus.cc:348] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.402093 8271 raft_consensus.cc:374] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:22.402122 8271 raft_consensus.cc:729] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:22.402238 8271 consensus_queue.cc:260] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.402282 8004 ts_tablet_manager.cc:1398] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:22.402514 8271 ts_tablet_manager.cc:1398] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:22.403096 8273 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:22.403184 8273 ts_tablet_manager.cc:1367] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:59:22.403645 8273 raft_consensus.cc:348] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:22.403738 8273 raft_consensus.cc:374] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:22.403772 8273 raft_consensus.cc:729] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:22.403893 8273 consensus_queue.cc:260] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:22.404152 8273 ts_tablet_manager.cc:1398] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:22.404314 8272 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:22.404408 8272 ts_tablet_manager.cc:1367] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:59:22.404839 8272 raft_consensus.cc:348] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:22.404933 8272 raft_consensus.cc:374] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:22.404971 8272 raft_consensus.cc:729] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:22.405087 8272 consensus_queue.cc:260] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:22.405344 8272 ts_tablet_manager.cc:1398] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:22.405509 8272 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:22.406391 8272 tablet_bootstrap.cc:654] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:22.407330 8272 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:22.407409 8272 ts_tablet_manager.cc:1367] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:22.407718 8272 raft_consensus.cc:348] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.407799 8272 raft_consensus.cc:374] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:22.407821 8272 raft_consensus.cc:729] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:22.407948 8272 consensus_queue.cc:260] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.408200 8272 ts_tablet_manager.cc:1398] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:22.421764 8058 raft_consensus.cc:482] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:22.421870 8058 raft_consensus.cc:504] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:22.422320 8058 leader_election.cc:288] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:22.422688 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e3c78c65334b52aca79d67c4bf45a2" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:22.422757 7180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e3c78c65334b52aca79d67c4bf45a2" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:22.422863 6917 raft_consensus.cc:2454] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:22.422895 7180 raft_consensus.cc:2454] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 0.
I20240502 01:59:22.423158 6985 leader_election.cc:302] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6101231591a458384f4ea6d1f0e0ae5, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:22.423410 8058 raft_consensus.cc:2781] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:22.423480 8058 raft_consensus.cc:482] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:22.423524 8058 raft_consensus.cc:3037] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:22.424391 8058 raft_consensus.cc:504] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:22.424729 8058 leader_election.cc:288] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Requested vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:22.425010 7180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e3c78c65334b52aca79d67c4bf45a2" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:22.425068 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e3c78c65334b52aca79d67c4bf45a2" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:22.425128 7180 raft_consensus.cc:3037] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:22.425182 6917 raft_consensus.cc:3037] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:22.425989 6917 raft_consensus.cc:2454] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:22.426127 7180 raft_consensus.cc:2454] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 1.
I20240502 01:59:22.426276 6985 leader_election.cc:302] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6101231591a458384f4ea6d1f0e0ae5, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:22.426478 8058 raft_consensus.cc:2781] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:22.426568 8058 raft_consensus.cc:686] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Becoming Leader. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Running, Role: LEADER
I20240502 01:59:22.426729 8058 consensus_queue.cc:237] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:22.428421 6641 catalog_manager.cc:5484] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 01:59:22.537333 8267 raft_consensus.cc:482] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:22.537464 8267 raft_consensus.cc:504] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.537866 8267 leader_election.cc:288] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:22.538261 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0548274c47148d9b611fd3c978a0c34" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 01:59:22.538321 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0548274c47148d9b611fd3c978a0c34" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:22.538429 7049 raft_consensus.cc:2454] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:22.538524 6917 raft_consensus.cc:2454] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:22.538753 6721 leader_election.cc:302] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:22.538949 8267 raft_consensus.cc:2781] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:22.538998 8267 raft_consensus.cc:482] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:22.539036 8267 raft_consensus.cc:3037] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:22.539877 8267 raft_consensus.cc:504] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.540175 8267 leader_election.cc:288] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Requested vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:22.540495 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0548274c47148d9b611fd3c978a0c34" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:22.540598 7049 raft_consensus.cc:3037] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:22.540557 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0548274c47148d9b611fd3c978a0c34" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:22.540665 6917 raft_consensus.cc:3037] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:22.541342 7049 raft_consensus.cc:2454] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 1.
I20240502 01:59:22.541433 6917 raft_consensus.cc:2454] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 1.
I20240502 01:59:22.541637 6721 leader_election.cc:302] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:22.541904 8267 raft_consensus.cc:2781] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:22.542052 8267 raft_consensus.cc:686] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Becoming Leader. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:22.542198 8267 consensus_queue.cc:237] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.544045 6641 catalog_manager.cc:5484] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: term changed from 0 to 1, leader changed from <none> to 85e78945910a4e1abfa101d2edc3644e (127.27.111.193). New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 01:59:22.696197 7836 raft_consensus.cc:482] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:22.696352 7836 raft_consensus.cc:504] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.696741 7836 leader_election.cc:288] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:22.697103 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c54b912f802461c8cac092ac3222c87" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:22.697183 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c54b912f802461c8cac092ac3222c87" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:22.697247 6785 raft_consensus.cc:2454] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 0.
I20240502 01:59:22.697329 6917 raft_consensus.cc:2454] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 0.
I20240502 01:59:22.697571 7116 leader_election.cc:302] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, 86bc8001c3464b929cd5c873861baa0c; no voters:
I20240502 01:59:22.697738 7836 raft_consensus.cc:2781] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:22.697798 7836 raft_consensus.cc:482] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:22.697824 7836 raft_consensus.cc:3037] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:22.698727 7836 raft_consensus.cc:504] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.699137 7836 leader_election.cc:288] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 election: Requested vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:22.699436 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c54b912f802461c8cac092ac3222c87" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:22.699491 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c54b912f802461c8cac092ac3222c87" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:22.699586 6917 raft_consensus.cc:3037] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:22.699589 6785 raft_consensus.cc:3037] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:22.700568 6785 raft_consensus.cc:2454] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 1.
I20240502 01:59:22.700568 6917 raft_consensus.cc:2454] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 1.
I20240502 01:59:22.700876 7116 leader_election.cc:302] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, 86bc8001c3464b929cd5c873861baa0c; no voters:
I20240502 01:59:22.701092 7836 raft_consensus.cc:2781] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:22.701166 7836 raft_consensus.cc:686] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Becoming Leader. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Running, Role: LEADER
I20240502 01:59:22.701319 7836 consensus_queue.cc:237] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:22.703152 6641 catalog_manager.cc:5484] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 86bc8001c3464b929cd5c873861baa0c (127.27.111.196). New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 01:59:22.809187 7792 consensus_queue.cc:1035] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:22.812553 7792 consensus_queue.cc:1035] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:59:22.868753 8286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:22.869056 8286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:22.872314 8286 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 01:59:22.960080 7049 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:22.960244 6917 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:22.960500 8267 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:22.960665 7832 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:22.966192 6785 raft_consensus.cc:1261] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:22.966382 6916 raft_consensus.cc:1261] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:22.966632 7836 consensus_queue.cc:1035] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:22.966782 8160 consensus_queue.cc:1035] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.27 ms
time per row: 0.0571352 ms
I20240502 01:59:23.029085 6641 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32904:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:59:23.034746 7030 tablet_service.cc:1467] Processing CreateTablet for tablet 91db5c1b3d2f4d22b5882a61ea0395c7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=19f17ecb48ac4307bcd5af6811f21abe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:23.035050 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91db5c1b3d2f4d22b5882a61ea0395c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:23.035190 7161 tablet_service.cc:1467] Processing CreateTablet for tablet 91db5c1b3d2f4d22b5882a61ea0395c7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=19f17ecb48ac4307bcd5af6811f21abe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:23.035466 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91db5c1b3d2f4d22b5882a61ea0395c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:23.037140 8347 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:23.037325 6765 tablet_service.cc:1467] Processing CreateTablet for tablet 91db5c1b3d2f4d22b5882a61ea0395c7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=19f17ecb48ac4307bcd5af6811f21abe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:23.037458 6764 tablet_service.cc:1467] Processing CreateTablet for tablet 4e44267bd266407f9b72c66e3d3579bb (DEFAULT_TABLE table=rebalance_test_table_9 [id=19f17ecb48ac4307bcd5af6811f21abe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:23.037576 6765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91db5c1b3d2f4d22b5882a61ea0395c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:23.037721 6764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e44267bd266407f9b72c66e3d3579bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:23.038297 8347 tablet_bootstrap.cc:654] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:23.039603 7159 tablet_service.cc:1467] Processing CreateTablet for tablet 347b161e9ec84640a0668836ce9e29eb (DEFAULT_TABLE table=rebalance_test_table_9 [id=19f17ecb48ac4307bcd5af6811f21abe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:23.039734 8348 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:23.039817 7159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 347b161e9ec84640a0668836ce9e29eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:23.040488 8348 tablet_bootstrap.cc:654] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:23.040478 6763 tablet_service.cc:1467] Processing CreateTablet for tablet 347b161e9ec84640a0668836ce9e29eb (DEFAULT_TABLE table=rebalance_test_table_9 [id=19f17ecb48ac4307bcd5af6811f21abe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:23.040726 6763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 347b161e9ec84640a0668836ce9e29eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:23.041504 8348 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:23.041572 8348 ts_tablet_manager.cc:1367] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:23.041951 8348 raft_consensus.cc:348] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:23.042069 8348 raft_consensus.cc:374] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:23.042109 8348 raft_consensus.cc:729] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:23.042204 8348 consensus_queue.cc:260] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:23.042539 8348 ts_tablet_manager.cc:1398] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:23.042699 7161 tablet_service.cc:1467] Processing CreateTablet for tablet 4e44267bd266407f9b72c66e3d3579bb (DEFAULT_TABLE table=rebalance_test_table_9 [id=19f17ecb48ac4307bcd5af6811f21abe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:23.042903 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e44267bd266407f9b72c66e3d3579bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:23.044327 6896 tablet_service.cc:1467] Processing CreateTablet for tablet 347b161e9ec84640a0668836ce9e29eb (DEFAULT_TABLE table=rebalance_test_table_9 [id=19f17ecb48ac4307bcd5af6811f21abe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:23.044591 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 347b161e9ec84640a0668836ce9e29eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:23.045256 8347 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:23.045334 8347 ts_tablet_manager.cc:1367] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:59:23.045770 8347 raft_consensus.cc:348] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:23.045883 8347 raft_consensus.cc:374] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:23.045920 8347 raft_consensus.cc:729] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:23.046028 8347 consensus_queue.cc:260] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:23.046304 8347 ts_tablet_manager.cc:1398] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:23.046738 8347 tablet_bootstrap.cc:492] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:23.047148 7030 tablet_service.cc:1467] Processing CreateTablet for tablet 4e44267bd266407f9b72c66e3d3579bb (DEFAULT_TABLE table=rebalance_test_table_9 [id=19f17ecb48ac4307bcd5af6811f21abe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:59:23.047381 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e44267bd266407f9b72c66e3d3579bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:23.047646 8347 tablet_bootstrap.cc:654] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:23.048681 8347 tablet_bootstrap.cc:492] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:23.048763 8347 ts_tablet_manager.cc:1367] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:23.049084 8347 raft_consensus.cc:348] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:23.049162 8347 raft_consensus.cc:374] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:23.049194 8347 raft_consensus.cc:729] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:23.049296 8347 consensus_queue.cc:260] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:23.049713 8347 ts_tablet_manager.cc:1398] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:23.049899 8347 tablet_bootstrap.cc:492] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:23.050747 8347 tablet_bootstrap.cc:654] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:23.050879 8348 tablet_bootstrap.cc:492] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:23.051740 8348 tablet_bootstrap.cc:654] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:23.052348 8347 tablet_bootstrap.cc:492] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c: No bootstrap required, opened a new log
I20240502 01:59:23.052459 8347 ts_tablet_manager.cc:1367] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:23.052474 8353 tablet_bootstrap.cc:492] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:23.052887 8347 raft_consensus.cc:348] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:23.052982 8347 raft_consensus.cc:374] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:23.053009 8347 raft_consensus.cc:729] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:23.053126 8347 consensus_queue.cc:260] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:23.053331 8353 tablet_bootstrap.cc:654] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:23.053375 8347 ts_tablet_manager.cc:1398] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:23.054507 8348 tablet_bootstrap.cc:492] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5: No bootstrap required, opened a new log
I20240502 01:59:23.054589 8348 ts_tablet_manager.cc:1367] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:59:23.054750 8353 tablet_bootstrap.cc:492] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8: No bootstrap required, opened a new log
I20240502 01:59:23.054838 8353 ts_tablet_manager.cc:1367] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:23.054993 8348 raft_consensus.cc:348] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:23.055107 8348 raft_consensus.cc:374] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:23.055137 8348 raft_consensus.cc:729] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:23.055245 8348 consensus_queue.cc:260] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:23.055298 8353 raft_consensus.cc:348] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:23.055403 8353 raft_consensus.cc:374] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:23.055433 8353 raft_consensus.cc:729] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:23.055486 8348 ts_tablet_manager.cc:1398] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:23.056128 8353 consensus_queue.cc:260] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:23.056455 8353 ts_tablet_manager.cc:1398] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:23.056612 8350 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:23.057603 8350 tablet_bootstrap.cc:654] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:23.058872 8350 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:23.058969 8350 ts_tablet_manager.cc:1367] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:23.059404 8350 raft_consensus.cc:348] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:23.059521 8350 raft_consensus.cc:374] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:23.059568 8350 raft_consensus.cc:729] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:23.059703 8350 consensus_queue.cc:260] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:23.059947 8350 ts_tablet_manager.cc:1398] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:23.060137 8350 tablet_bootstrap.cc:492] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:23.061224 8350 tablet_bootstrap.cc:654] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:23.062448 8350 tablet_bootstrap.cc:492] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:23.062554 8350 ts_tablet_manager.cc:1367] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:23.063056 8350 raft_consensus.cc:348] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:23.063164 8350 raft_consensus.cc:374] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:23.063207 8350 raft_consensus.cc:729] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:23.063344 8350 consensus_queue.cc:260] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:23.063629 8350 ts_tablet_manager.cc:1398] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:23.063783 8350 tablet_bootstrap.cc:492] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:23.064560 8350 tablet_bootstrap.cc:654] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e: Neither blocks nor log segments found. Creating new log.
I20240502 01:59:23.065920 8350 tablet_bootstrap.cc:492] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e: No bootstrap required, opened a new log
I20240502 01:59:23.065989 8350 ts_tablet_manager.cc:1367] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:23.066339 8350 raft_consensus.cc:348] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:23.066468 8350 raft_consensus.cc:374] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:59:23.066512 8350 raft_consensus.cc:729] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:23.066619 8350 consensus_queue.cc:260] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:23.066866 8350 ts_tablet_manager.cc:1398] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:23.195238 8336 raft_consensus.cc:482] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:23.195355 8336 raft_consensus.cc:504] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:23.195747 8336 leader_election.cc:288] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:23.196066 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91db5c1b3d2f4d22b5882a61ea0395c7" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 01:59:23.196066 7180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91db5c1b3d2f4d22b5882a61ea0395c7" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:23.196208 7180 raft_consensus.cc:2454] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:23.196208 7049 raft_consensus.cc:2454] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:23.196504 6721 leader_election.cc:302] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:23.196749 8336 raft_consensus.cc:2781] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:23.196811 8336 raft_consensus.cc:482] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:23.196867 8336 raft_consensus.cc:3037] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:23.197906 8336 raft_consensus.cc:504] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:23.198220 8336 leader_election.cc:288] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Requested vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:23.198554 7180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91db5c1b3d2f4d22b5882a61ea0395c7" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:23.198560 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91db5c1b3d2f4d22b5882a61ea0395c7" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:23.198652 7049 raft_consensus.cc:3037] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:23.198668 7180 raft_consensus.cc:3037] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:23.199386 7049 raft_consensus.cc:2454] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 1.
I20240502 01:59:23.199481 7180 raft_consensus.cc:2454] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 1.
I20240502 01:59:23.199666 6721 leader_election.cc:302] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:23.199824 8336 raft_consensus.cc:2781] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:23.199909 8336 raft_consensus.cc:686] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Becoming Leader. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:23.200057 8336 consensus_queue.cc:237] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:23.202035 6643 catalog_manager.cc:5484] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: term changed from 0 to 1, leader changed from <none> to 85e78945910a4e1abfa101d2edc3644e (127.27.111.193). New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:23.253695 8336 raft_consensus.cc:482] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:23.253860 8336 raft_consensus.cc:504] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:23.254249 8336 leader_election.cc:288] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:23.254735 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e44267bd266407f9b72c66e3d3579bb" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 01:59:23.254753 7180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e44267bd266407f9b72c66e3d3579bb" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
I20240502 01:59:23.254895 7180 raft_consensus.cc:2454] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:23.255213 7049 raft_consensus.cc:2454] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 0.
I20240502 01:59:23.255231 6720 leader_election.cc:302] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, 86bc8001c3464b929cd5c873861baa0c; no voters:
I20240502 01:59:23.255395 8336 raft_consensus.cc:2781] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:23.255445 8336 raft_consensus.cc:482] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:23.255482 8336 raft_consensus.cc:3037] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:23.256302 8336 raft_consensus.cc:504] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:23.256615 8336 leader_election.cc:288] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Requested vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:23.256984 7180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e44267bd266407f9b72c66e3d3579bb" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:23.256997 7049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e44267bd266407f9b72c66e3d3579bb" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:23.257081 7180 raft_consensus.cc:3037] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:23.257091 7049 raft_consensus.cc:3037] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:23.257786 7180 raft_consensus.cc:2454] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 1.
I20240502 01:59:23.257970 7049 raft_consensus.cc:2454] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 1.
I20240502 01:59:23.258098 6720 leader_election.cc:302] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, 86bc8001c3464b929cd5c873861baa0c; no voters:
I20240502 01:59:23.258289 8336 raft_consensus.cc:2781] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:23.258440 8336 raft_consensus.cc:686] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Becoming Leader. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:23.258584 8336 consensus_queue.cc:237] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:23.260601 6643 catalog_manager.cc:5484] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e reported cstate change: term changed from 0 to 1, leader changed from <none> to 85e78945910a4e1abfa101d2edc3644e (127.27.111.193). New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 01:59:23.281476 8160 raft_consensus.cc:482] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:59:23.281608 8160 raft_consensus.cc:504] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:23.281967 8160 leader_election.cc:288] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:23.282335 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347b161e9ec84640a0668836ce9e29eb" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e" is_pre_election: true
I20240502 01:59:23.282349 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347b161e9ec84640a0668836ce9e29eb" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" is_pre_election: true
I20240502 01:59:23.282509 6916 raft_consensus.cc:2454] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 0.
I20240502 01:59:23.282511 6785 raft_consensus.cc:2454] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 0.
I20240502 01:59:23.282821 7117 leader_election.cc:302] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:23.282999 8160 raft_consensus.cc:2781] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:59:23.283067 8160 raft_consensus.cc:482] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:59:23.283118 8160 raft_consensus.cc:3037] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:23.283932 8160 raft_consensus.cc:504] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:23.284222 8160 leader_election.cc:288] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 election: Requested vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:23.284518 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347b161e9ec84640a0668836ce9e29eb" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:23.284520 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "347b161e9ec84640a0668836ce9e29eb" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:23.284605 6916 raft_consensus.cc:3037] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:23.284616 6785 raft_consensus.cc:3037] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:59:23.285315 6916 raft_consensus.cc:2454] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 1.
I20240502 01:59:23.285403 6785 raft_consensus.cc:2454] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 1.
I20240502 01:59:23.285589 7117 leader_election.cc:302] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:23.285751 8160 raft_consensus.cc:2781] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:59:23.285831 8160 raft_consensus.cc:686] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Becoming Leader. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Running, Role: LEADER
I20240502 01:59:23.286173 8160 consensus_queue.cc:237] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:23.288064 6643 catalog_manager.cc:5484] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 86bc8001c3464b929cd5c873861baa0c (127.27.111.196). New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
W20240502 01:59:23.451355 8365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:23.451629 8365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:23.454826 8365 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 01:59:23.536330 7180 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:23.536798 8336 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:23.537928 7049 raft_consensus.cc:1261] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:23.538306 8267 consensus_queue.cc:1035] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:23.538705 7181 raft_consensus.cc:1261] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:23.539307 8267 consensus_queue.cc:1035] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:23.540292 6785 raft_consensus.cc:1261] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:23.540865 8160 consensus_queue.cc:1035] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:23.544198 7050 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:23.544662 8336 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:59:23.544726 6916 raft_consensus.cc:1261] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:59:23.545135 8160 consensus_queue.cc:1035] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 102.879 ms
time per row: 0.0514395 ms
I20240502 01:59:23.609561 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.197:45767
--local_ip_for_outbound_sockets=127.27.111.197
--tserver_master_addrs=127.27.111.254:44795
--webserver_port=38865
--webserver_interface=127.27.111.197
--builtin_ntp_servers=127.27.111.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:59:23.722651 8427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:23.722923 8427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:59:23.723014 8427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:23.725942 8427 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:59:23.726003 8427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:59:23.726106 8427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.197
I20240502 01:59:23.729694 8427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.197:45767
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.111.197
--webserver_port=38865
--tserver_master_addrs=127.27.111.254:44795
--never_fsync=true
--heap_profile_path=/tmp/kudu.8427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.197
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:59:23.730619 8427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:59:23.731729 8427 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:23.740876 8433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:23.741428 8435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:23.741972 8432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:23.742913 8427 server_base.cc:1034] running on GCE node
I20240502 01:59:23.743278 8427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:59:23.743803 8427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:59:23.744992 8427 hybrid_clock.cc:648] HybridClock initialized: now 1714615163744970 us; error 37 us; skew 500 ppm
I20240502 01:59:23.747081 8427 webserver.cc:460] Webserver started at http://127.27.111.197:38865/ using document root <none> and password file <none>
I20240502 01:59:23.747638 8427 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:23.747709 8427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:23.751051 8427 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 01:59:23.752909 8441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:23.753886 8427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240502 01:59:23.753988 8427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "954cd49326bf405abd81182d75dc783c"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:23.754427 8427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:23.767961 8427 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:23.768590 8427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:59:23.768736 8427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:23.769258 8427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:23.770328 8427 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:23.770424 8427 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:23.770483 8427 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:23.770510 8427 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:23.779435 8427 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.197:45767
I20240502 01:59:23.779453 8554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.197:45767 every 8 connection(s)
I20240502 01:59:23.780584 8427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:59:23.788740 8555 heartbeater.cc:347] Connected to a master server at 127.27.111.254:44795
I20240502 01:59:23.788821 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 8427
I20240502 01:59:23.789002 8555 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:23.789335 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.198:46475
--local_ip_for_outbound_sockets=127.27.111.198
--tserver_master_addrs=127.27.111.254:44795
--webserver_port=40441
--webserver_interface=127.27.111.198
--builtin_ntp_servers=127.27.111.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:59:23.789546 8555 heartbeater.cc:511] Master 127.27.111.254:44795 requested a full tablet report, sending...
I20240502 01:59:23.790753 6638 ts_manager.cc:194] Re-registered known tserver with Master: 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767)
I20240502 01:59:23.791344 6638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.197:48635
W20240502 01:59:23.899894 8558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:23.900117 8558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:59:23.900158 8558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:23.903209 8558 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:59:23.903270 8558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:59:23.903373 8558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.198
I20240502 01:59:23.907219 8558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.198:46475
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.27.111.198
--webserver_port=40441
--tserver_master_addrs=127.27.111.254:44795
--never_fsync=true
--heap_profile_path=/tmp/kudu.8558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.198
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:59:23.908072 8558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:59:23.909147 8558 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:23.917757 8565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:23.918543 8564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:23.919220 8567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:23.919888 8558 server_base.cc:1034] running on GCE node
I20240502 01:59:23.920305 8558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:59:23.920872 8558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:59:23.922014 8558 hybrid_clock.cc:648] HybridClock initialized: now 1714615163922007 us; error 27 us; skew 500 ppm
I20240502 01:59:23.924070 8558 webserver.cc:460] Webserver started at http://127.27.111.198:40441/ using document root <none> and password file <none>
I20240502 01:59:23.924657 8558 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:23.924741 8558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:23.928138 8558 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:59:23.930017 8573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:23.931103 8558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:23.931210 8558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4f53d80e39be410d92d4766692aff1c6"
format_stamp: "Formatted at 2024-05-02 01:59:13 on dist-test-slave-0pw1"
I20240502 01:59:23.931615 8558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:23.959149 8558 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:23.959851 8558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:59:23.960029 8558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:23.960618 8558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:23.961532 8558 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:23.961583 8558 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:23.961644 8558 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:23.961675 8558 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:23.970269 8558 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.198:46475
I20240502 01:59:23.970289 8687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.198:46475 every 8 connection(s)
I20240502 01:59:23.971382 8558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:59:23.975553 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 8558
I20240502 01:59:23.975983 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.199:41741
--local_ip_for_outbound_sockets=127.27.111.199
--tserver_master_addrs=127.27.111.254:44795
--webserver_port=45973
--webserver_interface=127.27.111.199
--builtin_ntp_servers=127.27.111.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:59:23.980310 8688 heartbeater.cc:347] Connected to a master server at 127.27.111.254:44795
I20240502 01:59:23.980568 8688 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:23.981197 8688 heartbeater.cc:511] Master 127.27.111.254:44795 requested a full tablet report, sending...
I20240502 01:59:23.982328 6638 ts_manager.cc:194] Re-registered known tserver with Master: 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
I20240502 01:59:23.983034 6638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.198:49501
W20240502 01:59:24.091984 8691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:24.092223 8691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:59:24.092293 8691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:24.095367 8691 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:59:24.095448 8691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:59:24.095542 8691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.199
I20240502 01:59:24.099157 8691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.199:41741
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.27.111.199
--webserver_port=45973
--tserver_master_addrs=127.27.111.254:44795
--never_fsync=true
--heap_profile_path=/tmp/kudu.8691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.199
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:59:24.100067 8691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:59:24.101197 8691 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:24.110080 8697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:24.110635 8699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:24.111174 8696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:24.112202 8691 server_base.cc:1034] running on GCE node
I20240502 01:59:24.112632 8691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:59:24.113237 8691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:59:24.114432 8691 hybrid_clock.cc:648] HybridClock initialized: now 1714615164114402 us; error 40 us; skew 500 ppm
I20240502 01:59:24.116727 8691 webserver.cc:460] Webserver started at http://127.27.111.199:45973/ using document root <none> and password file <none>
I20240502 01:59:24.117352 8691 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:24.117460 8691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:24.121173 8691 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:24.123133 8705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:24.124119 8691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:24.124225 8691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2"
format_stamp: "Formatted at 2024-05-02 01:59:14 on dist-test-slave-0pw1"
I20240502 01:59:24.124652 8691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:24.140291 8691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:24.140908 8691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:59:24.141064 8691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:24.141595 8691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:24.142521 8691 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:24.142570 8691 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:24.142628 8691 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:24.142657 8691 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:24.152377 8691 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.199:41741
I20240502 01:59:24.152398 8818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.199:41741 every 8 connection(s)
I20240502 01:59:24.153462 8691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:59:24.162262 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 8691
I20240502 01:59:24.162721 8819 heartbeater.cc:347] Connected to a master server at 127.27.111.254:44795
I20240502 01:59:24.162743 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.200:37235
--local_ip_for_outbound_sockets=127.27.111.200
--tserver_master_addrs=127.27.111.254:44795
--webserver_port=41449
--webserver_interface=127.27.111.200
--builtin_ntp_servers=127.27.111.212:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:59:24.162956 8819 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:24.163430 8819 heartbeater.cc:511] Master 127.27.111.254:44795 requested a full tablet report, sending...
I20240502 01:59:24.164382 6638 ts_manager.cc:194] Re-registered known tserver with Master: 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741)
I20240502 01:59:24.164966 6638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.199:60155
W20240502 01:59:24.277146 8822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:24.277386 8822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:59:24.277462 8822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:24.280539 8822 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:59:24.280618 8822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:59:24.280717 8822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.200
I20240502 01:59:24.284366 8822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.200:37235
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.27.111.200
--webserver_port=41449
--tserver_master_addrs=127.27.111.254:44795
--never_fsync=true
--heap_profile_path=/tmp/kudu.8822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.200
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 01:59:24.285356 8822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:59:24.286717 8822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:59:24.295547 8830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:24.296020 8827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:59:24.296624 8828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:59:24.297480 8822 server_base.cc:1034] running on GCE node
I20240502 01:59:24.297973 8822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:59:24.298607 8822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:59:24.299772 8822 hybrid_clock.cc:648] HybridClock initialized: now 1714615164299756 us; error 27 us; skew 500 ppm
I20240502 01:59:24.301818 8822 webserver.cc:460] Webserver started at http://127.27.111.200:41449/ using document root <none> and password file <none>
I20240502 01:59:24.302385 8822 fs_manager.cc:359] Metadata directory not provided
I20240502 01:59:24.302505 8822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:59:24.305892 8822 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:59:24.307854 8836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:24.308873 8822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:24.308979 8822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "070558c278e0457b9cc070ac0d662b5d"
format_stamp: "Formatted at 2024-05-02 01:59:14 on dist-test-slave-0pw1"
I20240502 01:59:24.309386 8822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:59:24.320528 8822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:59:24.321125 8822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:59:24.321288 8822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:59:24.321842 8822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:59:24.322750 8822 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:59:24.322824 8822 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:24.322881 8822 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:59:24.322916 8822 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:59:24.331756 8822 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.200:37235
I20240502 01:59:24.331812 8949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.200:37235 every 8 connection(s)
I20240502 01:59:24.332931 8822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:59:24.338864 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 8822
I20240502 01:59:24.343271 8950 heartbeater.cc:347] Connected to a master server at 127.27.111.254:44795
I20240502 01:59:24.343549 8950 heartbeater.cc:464] Registering TS with master...
I20240502 01:59:24.344182 8950 heartbeater.cc:511] Master 127.27.111.254:44795 requested a full tablet report, sending...
I20240502 01:59:24.345327 6638 ts_manager.cc:194] Re-registered known tserver with Master: 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
I20240502 01:59:24.346014 6638 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.200:50989
I20240502 01:59:24.753242 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:24.777920 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.782094 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.791761 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.793660 8555 heartbeater.cc:503] Master 127.27.111.254:44795 was elected leader, sending a full tablet report...
I20240502 01:59:24.801517 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.808570 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:24.810110 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:24.825287 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.833600 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:24.855086 6896 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:24.864749 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.874594 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:24.885303 8884 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.885594 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.887861 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:24.901211 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.914860 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.919694 6896 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:24.925527 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.936342 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:24.941686 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.947360 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.948477 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.963714 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:24.973263 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:24.975198 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.977125 6896 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:24.981019 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:24.981057 6896 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:24.981477 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:24.981727 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.985478 8688 heartbeater.cc:503] Master 127.27.111.254:44795 was elected leader, sending a full tablet report...
I20240502 01:59:24.989645 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.995790 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:24.998555 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.004513 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:25.006903 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.011907 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:25.016975 6896 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:25.044291 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.064314 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:25.167058 8819 heartbeater.cc:503] Master 127.27.111.254:44795 was elected leader, sending a full tablet report...
I20240502 01:59:25.297133 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.307386 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:25.332830 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.342339 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.347967 8950 heartbeater.cc:503] Master 127.27.111.254:44795 was elected leader, sending a full tablet report...
I20240502 01:59:25.348956 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:25.366050 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.371030 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.380347 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.383178 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.386466 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:25.391804 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.400651 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.404299 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:25.411818 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.416891 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.429471 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:25.430809 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:25.431283 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.433261 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:25.449609 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.462352 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.468466 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.473575 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.485769 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:25.486865 6896 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:25.488854 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.495175 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.510139 6896 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:25.510167 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:25.526916 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:25.527251 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:25.542272 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:25.546694 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:25.563324 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.572247 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.581202 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:25.602001 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:25.611490 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:25.628453 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:25.645109 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:25.822808 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.831413 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.832855 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.844007 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.851410 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:25.872373 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:25.897306 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.911259 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.915228 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:25.916525 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:25.926878 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.931342 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.935914 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:25.937074 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:25.940891 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.949007 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.951871 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.983067 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:25.984107 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:25.988023 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:25.994359 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:26.012210 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.014883 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.024077 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:26.025445 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.027065 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:26.030542 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.037391 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:26.061321 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.073163 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:26.099224 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:26.099552 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.107311 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:26.112072 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.120461 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:26.125085 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.141355 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:26.184252 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:26.186764 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.197181 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:26.257068 6917 consensus_queue.cc:237] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:26.259171 7181 raft_consensus.cc:1261] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.259620 7049 raft_consensus.cc:1261] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.259639 9616 consensus_queue.cc:1035] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.260025 9616 consensus_queue.cc:1035] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.264405 7181 consensus_queue.cc:237] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
W20240502 01:59:26.265295 6852 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17235e019e644b258266636fb6be6648. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.266165 7050 raft_consensus.cc:1261] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.266252 6783 raft_consensus.cc:1261] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.266661 8963 consensus_queue.cc:1035] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.266916 9658 consensus_queue.cc:1035] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.269692 9616 raft_consensus.cc:2932] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:26.272516 7181 raft_consensus.cc:2932] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:26.274286 7049 raft_consensus.cc:2932] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:26.277027 6635 catalog_manager.cc:5484] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
W20240502 01:59:26.279366 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4154be3f1f3c428580463c419d69892e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.279657 8963 raft_consensus.cc:2932] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.282158 7050 raft_consensus.cc:2932] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.285612 6783 raft_consensus.cc:2932] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.289104 6640 catalog_manager.cc:5484] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.292320 6917 consensus_queue.cc:237] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:26.294373 7050 raft_consensus.cc:1261] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.294857 9154 consensus_queue.cc:1035] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.295295 7181 raft_consensus.cc:1261] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.296443 9154 consensus_queue.cc:1035] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:59:26.299310 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed7b67af7b1c4255a7c2377570b58e70. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.301321 9756 raft_consensus.cc:2932] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:26.304598 7050 raft_consensus.cc:2932] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:26.308717 7181 raft_consensus.cc:2932] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:26.309055 6640 catalog_manager.cc:5484] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:26.326303 7050 consensus_queue.cc:237] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:26.328917 6917 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.329511 8330 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.330247 6783 consensus_queue.cc:237] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:26.330591 6785 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.331254 9685 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.332412 6916 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.332464 7049 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.336611 9588 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.337091 9588 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:59:26.339237 6985 consensus_peers.cc:485] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 034cf1c3d47a4f7785c16d16815c229d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.340870 7050 consensus_queue.cc:237] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:26.341199 6917 consensus_queue.cc:237] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:26.342813 6917 raft_consensus.cc:1261] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.343406 6785 raft_consensus.cc:1261] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.343667 6783 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:26.344131 7181 raft_consensus.cc:1261] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.344573 9756 consensus_queue.cc:1035] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.345621 8330 consensus_queue.cc:1035] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.346367 9685 consensus_queue.cc:1035] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.349728 6783 raft_consensus.cc:1261] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.349946 9843 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:26.350903 6635 catalog_manager.cc:5484] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
W20240502 01:59:26.352277 6852 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ecbf1c8ee8c46b195d2374a5db81ac0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.352851 9756 consensus_queue.cc:1035] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.354589 6916 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:26.360514 9685 raft_consensus.cc:2932] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.365074 6783 raft_consensus.cc:2932] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.370569 6637 catalog_manager.cc:5484] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.373426 6917 raft_consensus.cc:2932] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
W20240502 01:59:26.374029 6718 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0548274c47148d9b611fd3c978a0c34. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:59:26.375608 6985 consensus_peers.cc:485] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9313079c48c94ab1b7e022a91befafcd. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.378159 9756 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.381562 9848 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:26.381945 7049 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:26.381446 6637 catalog_manager.cc:5484] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.383989 6917 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:26.384086 7181 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.387748 6641 catalog_manager.cc:5484] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:26.391752 6783 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.423465 7049 consensus_queue.cc:237] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:26.425828 6783 raft_consensus.cc:1261] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.426316 9685 consensus_queue.cc:1035] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:59:26.426977 6985 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7345f51bd6f84b3880838c2460bafb75. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.428422 7181 raft_consensus.cc:1261] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.428989 9685 consensus_queue.cc:1035] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.429577 6917 consensus_queue.cc:237] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:26.432233 7180 raft_consensus.cc:1261] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:59:26.432627 6852 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17235e019e644b258266636fb6be6648. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.432845 9756 consensus_queue.cc:1035] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:59:26.433308 6852 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17235e019e644b258266636fb6be6648. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.433956 7049 raft_consensus.cc:1261] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:26.434540 9757 consensus_queue.cc:1035] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:26.436796 9756 raft_consensus.cc:2932] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.437263 7049 raft_consensus.cc:2932] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.439915 9685 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.440219 6783 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.440210 7180 raft_consensus.cc:2932] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.442005 6643 catalog_manager.cc:5484] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 4 to 5, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.446183 7180 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.447451 6636 catalog_manager.cc:5484] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:26.461112 6917 consensus_queue.cc:237] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:26.463451 6783 raft_consensus.cc:1261] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.463451 7180 raft_consensus.cc:1261] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.464186 9616 consensus_queue.cc:1035] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.464586 9757 consensus_queue.cc:1035] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.466509 9616 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.466989 6783 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
W20240502 01:59:26.469107 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39d9e48f67f34ebe850fc388ecd94ca3. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.469786 7180 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.470093 6640 catalog_manager.cc:5484] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.489401 7180 consensus_queue.cc:237] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:26.491683 6917 raft_consensus.cc:1261] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:59:26.492044 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 347b161e9ec84640a0668836ce9e29eb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.492156 9662 consensus_queue.cc:1035] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.492295 6783 raft_consensus.cc:1261] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.492820 9767 consensus_queue.cc:1035] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.494284 7049 consensus_queue.cc:237] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:26.497759 7180 raft_consensus.cc:1261] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:59:26.498531 6985 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7345f51bd6f84b3880838c2460bafb75. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.498631 6783 raft_consensus.cc:1261] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:26.499104 9860 consensus_queue.cc:1035] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:26.499426 9860 consensus_queue.cc:1035] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:26.503707 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.505753 9862 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.506891 9767 raft_consensus.cc:2932] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.508543 6783 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.509581 6639 catalog_manager.cc:5484] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.509526 6917 raft_consensus.cc:2932] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.510674 7180 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.512513 6642 catalog_manager.cc:5484] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 4 to 5, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.514768 6783 raft_consensus.cc:2932] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
W20240502 01:59:26.517541 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7345f51bd6f84b3880838c2460bafb75. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.522361 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.525540 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:26.528154 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.540244 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:26.560261 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.570174 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:26.571966 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:26.597504 6917 consensus_queue.cc:237] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:26.604290 7049 raft_consensus.cc:1261] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.603297 7180 raft_consensus.cc:1261] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:59:26.604971 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 818dbc9a8e054eeaa122e7919a311806. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.605036 9882 consensus_queue.cc:1035] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.605299 9616 consensus_queue.cc:1035] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.609474 7050 consensus_queue.cc:237] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:26.611225 7181 raft_consensus.cc:1261] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.611757 9843 consensus_queue.cc:1035] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.611857 6917 raft_consensus.cc:1261] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:59:26.612357 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14e3c78c65334b52aca79d67c4bf45a2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.613118 9843 consensus_queue.cc:1035] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.617408 9843 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:26.619446 6917 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:26.619881 7181 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:26.621814 6639 catalog_manager.cc:5484] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:26.628705 9882 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:26.630949 7049 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:26.631402 7180 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:26.633117 6642 catalog_manager.cc:5484] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:26.684069 6783 consensus_queue.cc:237] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:26.687462 6917 raft_consensus.cc:1261] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.687462 7049 raft_consensus.cc:1261] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.690623 9848 consensus_queue.cc:1035] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.690997 9848 consensus_queue.cc:1035] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:59:26.694305 6718 consensus_peers.cc:485] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f6c07e3923d47f29ef32561d7f438e1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.696528 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.697242 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.698853 6917 raft_consensus.cc:2932] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.701335 9883 raft_consensus.cc:2932] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.704128 7049 raft_consensus.cc:2932] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.705603 6636 catalog_manager.cc:5484] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:26.714402 7180 consensus_queue.cc:237] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:26.716406 6917 raft_consensus.cc:1261] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.716414 6785 raft_consensus.cc:1261] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:26.716881 8963 consensus_queue.cc:1035] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:26.719174 8963 consensus_queue.cc:1035] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:59:26.722092 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8529a93abd6e4d62bbc0bddcf2fda78a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:26.726239 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:26.728359 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.733115 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.741290 10170 ts_tablet_manager.cc:902] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:26.742195 10170 tablet_copy_client.cc:308] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 01:59:26.746304 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:26.752035 10184 ts_tablet_manager.cc:902] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:26.754554 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:26.764138 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:26.782891 10184 tablet_copy_client.cc:308] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:26.795536 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet ed7b67af7b1c4255a7c2377570b58e70 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 ({username='slave'} at 127.27.111.199:52619)
I20240502 01:59:26.795728 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet ed7b67af7b1c4255a7c2377570b58e70 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 at {username='slave'} at 127.27.111.199:52619: session id = 4e6e76b381af4cdc90bd3e9e6e8b5cf2-ed7b67af7b1c4255a7c2377570b58e70
I20240502 01:59:26.797264 6937 tablet_copy_source_session.cc:205] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:26.798672 8963 raft_consensus.cc:2932] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:26.803452 6917 raft_consensus.cc:2932] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:26.803650 6636 catalog_manager.cc:5484] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:26.804193 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed7b67af7b1c4255a7c2377570b58e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:26.808476 6785 raft_consensus.cc:2932] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:26.810106 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 9313079c48c94ab1b7e022a91befafcd from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:40005)
I20240502 01:59:26.810278 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 9313079c48c94ab1b7e022a91befafcd from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:40005: session id = 954cd49326bf405abd81182d75dc783c-9313079c48c94ab1b7e022a91befafcd
I20240502 01:59:26.811714 7070 tablet_copy_source_session.cc:205] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:26.816562 10170 tablet_copy_client.cc:783] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:26.816825 10170 tablet_copy_client.cc:647] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:26.817471 10184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9313079c48c94ab1b7e022a91befafcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:26.832950 10184 tablet_copy_client.cc:783] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:26.833323 10184 tablet_copy_client.cc:647] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:26.837101 10184 tablet_copy_client.cc:523] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:26.839278 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.840206 10184 tablet_bootstrap.cc:492] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 01:59:26.846585 10170 tablet_copy_client.cc:523] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:26.849748 10228 ts_tablet_manager.cc:902] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:26.850842 10228 tablet_copy_client.cc:308] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:36569
I20240502 01:59:26.851270 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.851399 10206 ts_tablet_manager.cc:902] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:26.851650 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:26.852353 10206 tablet_copy_client.cc:308] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:26.856463 10184 log.cc:824] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c: Log is configured to *not* fsync() on all Append() calls
I20240502 01:59:26.858990 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet a0548274c47148d9b611fd3c978a0c34 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 ({username='slave'} at 127.27.111.199:58595)
I20240502 01:59:26.859166 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet a0548274c47148d9b611fd3c978a0c34 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 at {username='slave'} at 127.27.111.199:58595: session id = 4e6e76b381af4cdc90bd3e9e6e8b5cf2-a0548274c47148d9b611fd3c978a0c34
I20240502 01:59:26.861184 6805 tablet_copy_source_session.cc:205] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:26.864717 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.867475 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0548274c47148d9b611fd3c978a0c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:26.870013 10206 tablet_copy_client.cc:783] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:26.870174 10206 tablet_copy_client.cc:647] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:26.871137 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:26.872714 10206 tablet_copy_client.cc:523] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:26.874964 10206 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap starting.
I20240502 01:59:26.881012 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.905679 10170 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap starting.
I20240502 01:59:26.908670 10257 ts_tablet_manager.cc:902] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:26.909989 10257 tablet_copy_client.cc:308] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:26.913976 10259 ts_tablet_manager.cc:902] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:26.915200 10259 tablet_copy_client.cc:308] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:36569
I20240502 01:59:26.916306 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:26.916483 7201 tablet_copy_service.cc:140] P 86bc8001c3464b929cd5c873861baa0c: Received BeginTabletCopySession request for tablet 347b161e9ec84640a0668836ce9e29eb from peer 070558c278e0457b9cc070ac0d662b5d ({username='slave'} at 127.27.111.200:56521)
I20240502 01:59:26.916492 7202 tablet_copy_service.cc:140] P 86bc8001c3464b929cd5c873861baa0c: Received BeginTabletCopySession request for tablet 4154be3f1f3c428580463c419d69892e from peer 070558c278e0457b9cc070ac0d662b5d ({username='slave'} at 127.27.111.200:56521)
I20240502 01:59:26.916680 7202 tablet_copy_service.cc:161] P 86bc8001c3464b929cd5c873861baa0c: Beginning new tablet copy session on tablet 4154be3f1f3c428580463c419d69892e from peer 070558c278e0457b9cc070ac0d662b5d at {username='slave'} at 127.27.111.200:56521: session id = 070558c278e0457b9cc070ac0d662b5d-4154be3f1f3c428580463c419d69892e
I20240502 01:59:26.917459 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:26.917447 10206 log.cc:824] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Log is configured to *not* fsync() on all Append() calls
I20240502 01:59:26.918740 7202 tablet_copy_source_session.cc:205] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:26.923063 10228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4154be3f1f3c428580463c419d69892e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:26.923101 10256 ts_tablet_manager.cc:902] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:26.923329 10258 ts_tablet_manager.cc:902] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:26.923861 10258 tablet_copy_client.cc:308] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 01:59:26.925945 10256 tablet_copy_client.cc:308] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 01:59:26.926607 7201 tablet_copy_service.cc:161] P 86bc8001c3464b929cd5c873861baa0c: Beginning new tablet copy session on tablet 347b161e9ec84640a0668836ce9e29eb from peer 070558c278e0457b9cc070ac0d662b5d at {username='slave'} at 127.27.111.200:56521: session id = 070558c278e0457b9cc070ac0d662b5d-347b161e9ec84640a0668836ce9e29eb
I20240502 01:59:26.927613 7201 tablet_copy_source_session.cc:205] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:26.928539 10259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 347b161e9ec84640a0668836ce9e29eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:26.934223 10259 tablet_copy_client.cc:783] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:26.934450 10259 tablet_copy_client.cc:647] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:26.936367 10259 tablet_copy_client.cc:523] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:26.938505 10259 tablet_bootstrap.cc:492] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d: Bootstrap starting.
I20240502 01:59:26.942859 10228 tablet_copy_client.cc:783] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:26.943068 10228 tablet_copy_client.cc:647] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:26.944636 10228 tablet_copy_client.cc:523] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:26.945910 10228 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d: Bootstrap starting.
I20240502 01:59:26.955103 10259 log.cc:824] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d: Log is configured to *not* fsync() on all Append() calls
I20240502 01:59:26.963546 10285 ts_tablet_manager.cc:902] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:26.963528 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet 39d9e48f67f34ebe850fc388ecd94ca3 from peer 070558c278e0457b9cc070ac0d662b5d ({username='slave'} at 127.27.111.200:54661)
I20240502 01:59:26.963651 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet 39d9e48f67f34ebe850fc388ecd94ca3 from peer 070558c278e0457b9cc070ac0d662b5d at {username='slave'} at 127.27.111.200:54661: session id = 070558c278e0457b9cc070ac0d662b5d-39d9e48f67f34ebe850fc388ecd94ca3
I20240502 01:59:26.964313 10285 tablet_copy_client.cc:308] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 01:59:26.964784 6937 tablet_copy_source_session.cc:205] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:26.965901 10258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d9e48f67f34ebe850fc388ecd94ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:26.968252 10258 tablet_copy_client.cc:783] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:26.968405 10258 tablet_copy_client.cc:647] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:26.971725 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:47485)
I20240502 01:59:26.971825 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:47485: session id = 954cd49326bf405abd81182d75dc783c-9ecbf1c8ee8c46b195d2374a5db81ac0
I20240502 01:59:26.972707 6937 tablet_copy_source_session.cc:205] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:26.974965 10258 tablet_copy_client.cc:523] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:26.976469 10258 tablet_bootstrap.cc:492] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap starting.
I20240502 01:59:26.982036 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:26.984794 10206 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:26.989289 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.989984 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 034cf1c3d47a4f7785c16d16815c229d from peer 4f53d80e39be410d92d4766692aff1c6 ({username='slave'} at 127.27.111.198:57069)
I20240502 01:59:26.990092 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 034cf1c3d47a4f7785c16d16815c229d from peer 4f53d80e39be410d92d4766692aff1c6 at {username='slave'} at 127.27.111.198:57069: session id = 4f53d80e39be410d92d4766692aff1c6-034cf1c3d47a4f7785c16d16815c229d
I20240502 01:59:26.990661 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet 17235e019e644b258266636fb6be6648 from peer 4f53d80e39be410d92d4766692aff1c6 ({username='slave'} at 127.27.111.198:35861)
I20240502 01:59:26.990772 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet 17235e019e644b258266636fb6be6648 from peer 4f53d80e39be410d92d4766692aff1c6 at {username='slave'} at 127.27.111.198:35861: session id = 4f53d80e39be410d92d4766692aff1c6-17235e019e644b258266636fb6be6648
I20240502 01:59:26.991178 7070 tablet_copy_source_session.cc:205] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:26.991663 6937 tablet_copy_source_session.cc:205] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:26.995093 10256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17235e019e644b258266636fb6be6648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:26.995283 10285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:26.997807 10285 tablet_copy_client.cc:783] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:26.997972 10285 tablet_copy_client.cc:647] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:27.008790 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034cf1c3d47a4f7785c16d16815c229d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:27.010727 10285 tablet_copy_client.cc:523] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:27.012477 10285 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 01:59:27.016222 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.022132 10305 ts_tablet_manager.cc:902] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:27.025029 10206 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap complete.
I20240502 01:59:27.025439 10206 ts_tablet_manager.cc:1367] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent bootstrapping tablet: real 0.151s user 0.039s sys 0.000s
I20240502 01:59:27.038935 10305 tablet_copy_client.cc:308] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 01:59:27.039577 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet 17235e019e644b258266636fb6be6648 from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:47485)
I20240502 01:59:27.039693 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet 17235e019e644b258266636fb6be6648 from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:47485: session id = 954cd49326bf405abd81182d75dc783c-17235e019e644b258266636fb6be6648
I20240502 01:59:27.040644 6937 tablet_copy_source_session.cc:205] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:27.041692 10305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17235e019e644b258266636fb6be6648. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:27.044618 10305 tablet_copy_client.cc:783] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:27.044849 10305 tablet_copy_client.cc:647] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:27.048611 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.050182 10170 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.050546 10170 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap complete.
I20240502 01:59:27.050804 10170 ts_tablet_manager.cc:1367] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent bootstrapping tablet: real 0.146s user 0.028s sys 0.001s
I20240502 01:59:27.050617 10257 tablet_copy_client.cc:783] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:27.051496 10257 tablet_copy_client.cc:647] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:27.053328 10257 tablet_copy_client.cc:523] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:27.054139 10170 raft_consensus.cc:348] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:27.055524 10303 ts_tablet_manager.cc:902] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:27.056033 10303 tablet_copy_client.cc:308] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:27.056531 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 14e3c78c65334b52aca79d67c4bf45a2 from peer 4f53d80e39be410d92d4766692aff1c6 ({username='slave'} at 127.27.111.198:57069)
I20240502 01:59:27.056631 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 14e3c78c65334b52aca79d67c4bf45a2 from peer 4f53d80e39be410d92d4766692aff1c6 at {username='slave'} at 127.27.111.198:57069: session id = 4f53d80e39be410d92d4766692aff1c6-14e3c78c65334b52aca79d67c4bf45a2
I20240502 01:59:27.057658 7070 tablet_copy_source_session.cc:205] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:27.058193 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:27.058549 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.058962 10303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e3c78c65334b52aca79d67c4bf45a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:27.059089 10305 tablet_copy_client.cc:523] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:27.061964 10303 tablet_copy_client.cc:783] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:27.062129 10303 tablet_copy_client.cc:647] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:27.064693 10305 tablet_bootstrap.cc:492] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 01:59:27.064931 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:27.068375 10256 tablet_copy_client.cc:783] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:27.068696 10256 tablet_copy_client.cc:647] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:27.070336 10256 tablet_copy_client.cc:523] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:27.072270 10256 tablet_bootstrap.cc:492] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap starting.
I20240502 01:59:27.074820 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:27.077190 10315 ts_tablet_manager.cc:902] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:27.077950 10315 tablet_copy_client.cc:308] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:27.082656 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 7345f51bd6f84b3880838c2460bafb75 from peer 070558c278e0457b9cc070ac0d662b5d ({username='slave'} at 127.27.111.200:49029)
I20240502 01:59:27.082775 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 7345f51bd6f84b3880838c2460bafb75 from peer 070558c278e0457b9cc070ac0d662b5d at {username='slave'} at 127.27.111.200:49029: session id = 070558c278e0457b9cc070ac0d662b5d-7345f51bd6f84b3880838c2460bafb75
I20240502 01:59:27.083781 7070 tablet_copy_source_session.cc:205] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:27.086114 10303 tablet_copy_client.cc:523] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:27.088042 10303 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap starting.
I20240502 01:59:27.090559 10170 raft_consensus.cc:729] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.076190 10206 raft_consensus.cc:348] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:27.090907 10206 raft_consensus.cc:729] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.091347 10206 consensus_queue.cc:260] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:27.092408 10206 ts_tablet_manager.cc:1398] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent starting tablet: real 0.067s user 0.002s sys 0.000s
I20240502 01:59:27.093323 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-a0548274c47148d9b611fd3c978a0c34 received from {username='slave'} at 127.27.111.199:58595
I20240502 01:59:27.093415 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-a0548274c47148d9b611fd3c978a0c34 on tablet a0548274c47148d9b611fd3c978a0c34 with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:27.094131 10170 consensus_queue.cc:260] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:27.094856 10170 ts_tablet_manager.cc:1398] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent starting tablet: real 0.044s user 0.000s sys 0.004s
I20240502 01:59:27.095350 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-ed7b67af7b1c4255a7c2377570b58e70 received from {username='slave'} at 127.27.111.199:52619
I20240502 01:59:27.095436 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-ed7b67af7b1c4255a7c2377570b58e70 on tablet ed7b67af7b1c4255a7c2377570b58e70 with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:27.114837 10315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7345f51bd6f84b3880838c2460bafb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:27.118098 10315 tablet_copy_client.cc:783] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:27.118532 10257 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6: Bootstrap starting.
I20240502 01:59:27.125685 10256 log.cc:824] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6: Log is configured to *not* fsync() on all Append() calls
I20240502 01:59:27.118873 10228 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.132602 10228 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d: Bootstrap complete.
I20240502 01:59:27.133078 10228 ts_tablet_manager.cc:1367] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d: Time spent bootstrapping tablet: real 0.187s user 0.025s sys 0.007s
I20240502 01:59:27.135988 10228 raft_consensus.cc:348] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:27.131377 10315 tablet_copy_client.cc:647] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:27.137936 10315 tablet_copy_client.cc:523] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:27.139107 10285 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.139492 10285 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
I20240502 01:59:27.139894 10315 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap starting.
I20240502 01:59:27.139926 10285 ts_tablet_manager.cc:1367] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.127s user 0.032s sys 0.000s
I20240502 01:59:27.142928 10285 raft_consensus.cc:348] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:27.129948 10259 tablet_bootstrap.cc:492] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.143242 10285 raft_consensus.cc:729] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.136430 10228 raft_consensus.cc:729] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.143730 10259 tablet_bootstrap.cc:492] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d: Bootstrap complete.
I20240502 01:59:27.143993 10259 ts_tablet_manager.cc:1367] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d: Time spent bootstrapping tablet: real 0.206s user 0.036s sys 0.001s
I20240502 01:59:27.143945 10228 consensus_queue.cc:260] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:27.145087 10228 ts_tablet_manager.cc:1398] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240502 01:59:27.145828 10259 raft_consensus.cc:348] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:27.146677 10259 raft_consensus.cc:729] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.146863 10259 consensus_queue.cc:260] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:27.149041 10285 consensus_queue.cc:260] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:27.149605 10259 ts_tablet_manager.cc:1398] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:59:27.150065 10285 ts_tablet_manager.cc:1398] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20240502 01:59:27.150640 7201 tablet_copy_service.cc:342] P 86bc8001c3464b929cd5c873861baa0c: Request end of tablet copy session 070558c278e0457b9cc070ac0d662b5d-4154be3f1f3c428580463c419d69892e received from {username='slave'} at 127.27.111.200:56521
I20240502 01:59:27.150763 7201 tablet_copy_service.cc:433] P 86bc8001c3464b929cd5c873861baa0c: ending tablet copy session 070558c278e0457b9cc070ac0d662b5d-4154be3f1f3c428580463c419d69892e on tablet 4154be3f1f3c428580463c419d69892e with peer 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:27.152597 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-9ecbf1c8ee8c46b195d2374a5db81ac0 received from {username='slave'} at 127.27.111.197:47485
I20240502 01:59:27.152695 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 954cd49326bf405abd81182d75dc783c-9ecbf1c8ee8c46b195d2374a5db81ac0 on tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 with peer 954cd49326bf405abd81182d75dc783c
I20240502 01:59:27.153883 10285 ts_tablet_manager.cc:902] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:27.154304 10285 tablet_copy_client.cc:308] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:27.154793 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 7345f51bd6f84b3880838c2460bafb75 from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:40005)
I20240502 01:59:27.154927 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 7345f51bd6f84b3880838c2460bafb75 from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:40005: session id = 954cd49326bf405abd81182d75dc783c-7345f51bd6f84b3880838c2460bafb75
I20240502 01:59:27.155994 7070 tablet_copy_source_session.cc:205] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:27.157384 10285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7345f51bd6f84b3880838c2460bafb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:27.158622 7201 tablet_copy_service.cc:342] P 86bc8001c3464b929cd5c873861baa0c: Request end of tablet copy session 070558c278e0457b9cc070ac0d662b5d-347b161e9ec84640a0668836ce9e29eb received from {username='slave'} at 127.27.111.200:56521
I20240502 01:59:27.158721 7201 tablet_copy_service.cc:433] P 86bc8001c3464b929cd5c873861baa0c: ending tablet copy session 070558c278e0457b9cc070ac0d662b5d-347b161e9ec84640a0668836ce9e29eb on tablet 347b161e9ec84640a0668836ce9e29eb with peer 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:27.161578 10285 tablet_copy_client.cc:783] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:27.161768 10285 tablet_copy_client.cc:647] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:27.162983 10285 tablet_copy_client.cc:523] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:27.164374 10285 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 01:59:27.168560 10360 ts_tablet_manager.cc:902] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:27.170094 10360 tablet_copy_client.cc:308] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 01:59:27.176923 10305 tablet_bootstrap.cc:492] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.177392 10305 tablet_bootstrap.cc:492] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
I20240502 01:59:27.177649 10305 ts_tablet_manager.cc:1367] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.113s user 0.029s sys 0.000s
I20240502 01:59:27.178360 10305 raft_consensus.cc:348] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:27.179730 10305 raft_consensus.cc:729] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.179981 10305 consensus_queue.cc:260] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:27.180426 10305 ts_tablet_manager.cc:1398] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:27.181362 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet 818dbc9a8e054eeaa122e7919a311806 from peer 4f53d80e39be410d92d4766692aff1c6 ({username='slave'} at 127.27.111.198:35861)
I20240502 01:59:27.181509 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet 818dbc9a8e054eeaa122e7919a311806 from peer 4f53d80e39be410d92d4766692aff1c6 at {username='slave'} at 127.27.111.198:35861: session id = 4f53d80e39be410d92d4766692aff1c6-818dbc9a8e054eeaa122e7919a311806
I20240502 01:59:27.182633 6937 tablet_copy_source_session.cc:205] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:27.183200 7180 consensus_queue.cc:237] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:27.183560 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-17235e019e644b258266636fb6be6648 received from {username='slave'} at 127.27.111.197:47485
I20240502 01:59:27.183642 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 954cd49326bf405abd81182d75dc783c-17235e019e644b258266636fb6be6648 on tablet 17235e019e644b258266636fb6be6648 with peer 954cd49326bf405abd81182d75dc783c
I20240502 01:59:27.184473 10360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818dbc9a8e054eeaa122e7919a311806. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:27.184733 7049 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.185364 6785 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.185983 9767 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.186348 9767 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:59:27.186674 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6ac26fa6de44f7ca68e6c0b6b959db7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.187151 10360 tablet_copy_client.cc:783] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:27.187321 10360 tablet_copy_client.cc:647] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:27.191087 10360 tablet_copy_client.cc:523] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:27.192625 10360 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap starting.
I20240502 01:59:27.202878 10303 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.203336 10303 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap complete.
I20240502 01:59:27.203727 10303 ts_tablet_manager.cc:1367] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6: Time spent bootstrapping tablet: real 0.116s user 0.038s sys 0.000s
I20240502 01:59:27.204188 9662 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.206866 7049 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.207134 10303 raft_consensus.cc:348] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:27.207398 10303 raft_consensus.cc:729] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.207929 10303 consensus_queue.cc:260] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:27.209739 6643 catalog_manager.cc:5484] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.210942 6785 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.213145 10303 ts_tablet_manager.cc:1398] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240502 01:59:27.217003 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 4f53d80e39be410d92d4766692aff1c6-14e3c78c65334b52aca79d67c4bf45a2 received from {username='slave'} at 127.27.111.198:57069
I20240502 01:59:27.217108 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 4f53d80e39be410d92d4766692aff1c6-14e3c78c65334b52aca79d67c4bf45a2 on tablet 14e3c78c65334b52aca79d67c4bf45a2 with peer 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:27.219573 8757 raft_consensus.cc:1203] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:27.223486 6785 consensus_queue.cc:237] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:27.225138 7180 raft_consensus.cc:1261] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.225425 6917 raft_consensus.cc:1261] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.225826 9848 consensus_queue.cc:1035] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.226171 9848 consensus_queue.cc:1035] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.227281 10285 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.227614 10285 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
W20240502 01:59:27.227595 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4fb9af11abc49d496b5c3d8048c22c8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.227838 10285 ts_tablet_manager.cc:1367] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.064s user 0.024s sys 0.004s
I20240502 01:59:27.228358 10285 raft_consensus.cc:348] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:27.228492 10285 raft_consensus.cc:729] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.228667 10285 consensus_queue.cc:260] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:27.229063 10285 ts_tablet_manager.cc:1398] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:27.229574 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-7345f51bd6f84b3880838c2460bafb75 received from {username='slave'} at 127.27.111.197:40005
I20240502 01:59:27.229655 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 954cd49326bf405abd81182d75dc783c-7345f51bd6f84b3880838c2460bafb75 on tablet 7345f51bd6f84b3880838c2460bafb75 with peer 954cd49326bf405abd81182d75dc783c
I20240502 01:59:27.236205 9848 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.238811 7180 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.238979 6917 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.242341 6634 catalog_manager.cc:5484] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.250113 10259 ts_tablet_manager.cc:902] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:27.250612 10259 tablet_copy_client.cc:308] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:27.255023 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet 9f6c07e3923d47f29ef32561d7f438e1 from peer 070558c278e0457b9cc070ac0d662b5d ({username='slave'} at 127.27.111.200:36335)
I20240502 01:59:27.255142 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet 9f6c07e3923d47f29ef32561d7f438e1 from peer 070558c278e0457b9cc070ac0d662b5d at {username='slave'} at 127.27.111.200:36335: session id = 070558c278e0457b9cc070ac0d662b5d-9f6c07e3923d47f29ef32561d7f438e1
I20240502 01:59:27.255368 10184 tablet_bootstrap.cc:492] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.255713 10184 tablet_bootstrap.cc:492] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
I20240502 01:59:27.255905 10184 ts_tablet_manager.cc:1367] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.416s user 0.036s sys 0.003s
I20240502 01:59:27.256088 6805 tablet_copy_source_session.cc:205] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:27.256366 10184 raft_consensus.cc:348] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:27.256484 10184 raft_consensus.cc:729] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.256713 10184 consensus_queue.cc:260] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:27.257411 10259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f6c07e3923d47f29ef32561d7f438e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:27.259708 10259 tablet_copy_client.cc:783] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:27.259864 10259 tablet_copy_client.cc:647] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:27.267261 10259 tablet_copy_client.cc:523] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:27.268735 10259 tablet_bootstrap.cc:492] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap starting.
I20240502 01:59:27.283828 10184 ts_tablet_manager.cc:1398] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240502 01:59:27.284739 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-9313079c48c94ab1b7e022a91befafcd received from {username='slave'} at 127.27.111.197:40005
I20240502 01:59:27.284834 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 954cd49326bf405abd81182d75dc783c-9313079c48c94ab1b7e022a91befafcd on tablet 9313079c48c94ab1b7e022a91befafcd with peer 954cd49326bf405abd81182d75dc783c
I20240502 01:59:27.292944 10256 tablet_bootstrap.cc:492] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.294997 10257 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.295266 10256 tablet_bootstrap.cc:492] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap complete.
I20240502 01:59:27.295480 10256 ts_tablet_manager.cc:1367] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6: Time spent bootstrapping tablet: real 0.223s user 0.032s sys 0.002s
I20240502 01:59:27.296195 10256 raft_consensus.cc:348] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:27.296343 10256 raft_consensus.cc:729] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.299057 10315 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.299470 10257 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6: Bootstrap complete.
I20240502 01:59:27.299942 10257 ts_tablet_manager.cc:1367] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6: Time spent bootstrapping tablet: real 0.181s user 0.023s sys 0.010s
I20240502 01:59:27.299628 10256 consensus_queue.cc:260] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:27.301856 10256 ts_tablet_manager.cc:1398] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:59:27.302423 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 4f53d80e39be410d92d4766692aff1c6-17235e019e644b258266636fb6be6648 received from {username='slave'} at 127.27.111.198:35861
I20240502 01:59:27.302515 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 4f53d80e39be410d92d4766692aff1c6-17235e019e644b258266636fb6be6648 on tablet 17235e019e644b258266636fb6be6648 with peer 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:27.304610 10257 raft_consensus.cc:348] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:27.304836 10257 raft_consensus.cc:729] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.305708 10257 consensus_queue.cc:260] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:27.307660 10257 ts_tablet_manager.cc:1398] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 01:59:27.313867 10170 ts_tablet_manager.cc:902] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:27.314436 10170 tablet_copy_client.cc:308] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:36569
I20240502 01:59:27.303959 8903 raft_consensus.cc:1203] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:27.316609 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 4f53d80e39be410d92d4766692aff1c6-034cf1c3d47a4f7785c16d16815c229d received from {username='slave'} at 127.27.111.198:57069
I20240502 01:59:27.316711 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 4f53d80e39be410d92d4766692aff1c6-034cf1c3d47a4f7785c16d16815c229d on tablet 034cf1c3d47a4f7785c16d16815c229d with peer 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:27.307519 10259 tablet_bootstrap.cc:492] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.319173 7201 tablet_copy_service.cc:140] P 86bc8001c3464b929cd5c873861baa0c: Received BeginTabletCopySession request for tablet 8529a93abd6e4d62bbc0bddcf2fda78a from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 ({username='slave'} at 127.27.111.199:47693)
I20240502 01:59:27.319283 7201 tablet_copy_service.cc:161] P 86bc8001c3464b929cd5c873861baa0c: Beginning new tablet copy session on tablet 8529a93abd6e4d62bbc0bddcf2fda78a from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 at {username='slave'} at 127.27.111.199:47693: session id = 4e6e76b381af4cdc90bd3e9e6e8b5cf2-8529a93abd6e4d62bbc0bddcf2fda78a
I20240502 01:59:27.319465 10259 tablet_bootstrap.cc:492] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap complete.
I20240502 01:59:27.319669 10259 ts_tablet_manager.cc:1367] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d: Time spent bootstrapping tablet: real 0.051s user 0.029s sys 0.001s
I20240502 01:59:27.320273 7201 tablet_copy_source_session.cc:205] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:27.320384 10259 raft_consensus.cc:348] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:27.321108 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8529a93abd6e4d62bbc0bddcf2fda78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:27.322019 10315 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap complete.
I20240502 01:59:27.322191 8757 raft_consensus.cc:1203] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:27.322418 10315 ts_tablet_manager.cc:1367] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d: Time spent bootstrapping tablet: real 0.183s user 0.030s sys 0.000s
I20240502 01:59:27.323048 10315 raft_consensus.cc:348] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:27.323175 10315 raft_consensus.cc:729] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.323503 10170 tablet_copy_client.cc:783] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:27.323582 10315 consensus_queue.cc:260] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:27.323655 10170 tablet_copy_client.cc:647] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:27.313129 10258 tablet_bootstrap.cc:492] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.325129 10258 tablet_bootstrap.cc:492] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap complete.
I20240502 01:59:27.325311 10258 ts_tablet_manager.cc:1367] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d: Time spent bootstrapping tablet: real 0.349s user 0.027s sys 0.004s
I20240502 01:59:27.325966 10315 ts_tablet_manager.cc:1398] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:27.326493 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 070558c278e0457b9cc070ac0d662b5d-7345f51bd6f84b3880838c2460bafb75 received from {username='slave'} at 127.27.111.200:49029
I20240502 01:59:27.320504 10259 raft_consensus.cc:729] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.326582 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 070558c278e0457b9cc070ac0d662b5d-7345f51bd6f84b3880838c2460bafb75 on tablet 7345f51bd6f84b3880838c2460bafb75 with peer 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:27.326686 10259 consensus_queue.cc:260] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:27.327126 10259 ts_tablet_manager.cc:1398] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:59:27.327490 10258 raft_consensus.cc:348] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:27.327683 10258 raft_consensus.cc:729] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.327680 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 070558c278e0457b9cc070ac0d662b5d-9f6c07e3923d47f29ef32561d7f438e1 received from {username='slave'} at 127.27.111.200:36335
I20240502 01:59:27.327764 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 070558c278e0457b9cc070ac0d662b5d-9f6c07e3923d47f29ef32561d7f438e1 on tablet 9f6c07e3923d47f29ef32561d7f438e1 with peer 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:27.327967 10258 consensus_queue.cc:260] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:27.328506 10258 ts_tablet_manager.cc:1398] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:27.329231 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 070558c278e0457b9cc070ac0d662b5d-39d9e48f67f34ebe850fc388ecd94ca3 received from {username='slave'} at 127.27.111.200:54661
I20240502 01:59:27.329344 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 070558c278e0457b9cc070ac0d662b5d-39d9e48f67f34ebe850fc388ecd94ca3 on tablet 39d9e48f67f34ebe850fc388ecd94ca3 with peer 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:27.338528 10170 tablet_copy_client.cc:523] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:27.338922 10360 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.339238 10360 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap complete.
I20240502 01:59:27.339440 10360 ts_tablet_manager.cc:1367] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6: Time spent bootstrapping tablet: real 0.147s user 0.027s sys 0.000s
I20240502 01:59:27.340190 10170 tablet_bootstrap.cc:492] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap starting.
I20240502 01:59:27.342067 10360 raft_consensus.cc:348] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:27.342330 10360 raft_consensus.cc:729] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.343657 10360 consensus_queue.cc:260] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:27.345482 10360 ts_tablet_manager.cc:1398] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:59:27.346082 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 4f53d80e39be410d92d4766692aff1c6-818dbc9a8e054eeaa122e7919a311806 received from {username='slave'} at 127.27.111.198:35861
I20240502 01:59:27.346174 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 4f53d80e39be410d92d4766692aff1c6-818dbc9a8e054eeaa122e7919a311806 on tablet 818dbc9a8e054eeaa122e7919a311806 with peer 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:27.350776 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.365164 7049 consensus_queue.cc:237] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:27.367138 6785 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.368538 8641 raft_consensus.cc:1203] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:59:27.369289 7180 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.369553 9843 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.371213 9862 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.372995 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.377254 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.380689 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:27.391860 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:27.394773 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:27.395462 8903 raft_consensus.cc:1203] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:27.399973 8903 raft_consensus.cc:1203] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:27.404150 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:27.409101 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48eeb721a60c4be888041bd31cfeab7a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.418938 8509 raft_consensus.cc:1203] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:27.438824 8509 raft_consensus.cc:1203] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:59:27.443089 7049 consensus_queue.cc:237] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
W20240502 01:59:27.444998 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47a276e9780e448a9b3c6eab67925514. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.445008 6785 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.445504 9862 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.447141 7181 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.447647 9862 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.449947 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.458170 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:27.458892 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:27.459547 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.463272 9843 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.465546 6785 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.465641 7180 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.467774 6635 catalog_manager.cc:5484] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.468649 9843 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.469722 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:27.472254 6783 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.475327 8641 raft_consensus.cc:1203] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:27.475564 6637 catalog_manager.cc:5484] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.479712 7181 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.481724 8903 raft_consensus.cc:1203] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:59:27.482609 10170 tablet_bootstrap.cc:492] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.482924 10170 tablet_bootstrap.cc:492] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap complete.
I20240502 01:59:27.485877 10170 ts_tablet_manager.cc:1367] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent bootstrapping tablet: real 0.146s user 0.030s sys 0.000s
I20240502 01:59:27.486362 10170 raft_consensus.cc:348] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:27.486502 10170 raft_consensus.cc:729] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.486531 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.486660 10170 consensus_queue.cc:260] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:27.487303 10170 ts_tablet_manager.cc:1398] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:27.489691 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.491303 8641 raft_consensus.cc:1203] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:27.491312 7201 tablet_copy_service.cc:342] P 86bc8001c3464b929cd5c873861baa0c: Request end of tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-8529a93abd6e4d62bbc0bddcf2fda78a received from {username='slave'} at 127.27.111.199:47693
I20240502 01:59:27.491393 7201 tablet_copy_service.cc:433] P 86bc8001c3464b929cd5c873861baa0c: ending tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-8529a93abd6e4d62bbc0bddcf2fda78a on tablet 8529a93abd6e4d62bbc0bddcf2fda78a with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:27.493775 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:27.507210 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:27.559125 6783 consensus_queue.cc:237] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:27.564218 7181 raft_consensus.cc:1261] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.564694 10394 consensus_queue.cc:1035] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.566376 6917 raft_consensus.cc:1261] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.567047 10394 consensus_queue.cc:1035] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.575420 10394 raft_consensus.cc:2932] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.577778 7181 raft_consensus.cc:2932] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.579988 6917 raft_consensus.cc:2932] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
W20240502 01:59:27.580157 6721 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a66b932d884e4df8a78841103d88ded0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.582126 6637 catalog_manager.cc:5484] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.584360 6783 consensus_queue.cc:237] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:27.586907 7049 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:59:27.587744 6721 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91db5c1b3d2f4d22b5882a61ea0395c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.588085 9848 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.586372 7181 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.588928 9848 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.589449 7049 consensus_queue.cc:237] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:27.590998 6783 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:27.591450 7182 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:27.591876 10486 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:27.592022 9862 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:59:27.592532 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48eeb721a60c4be888041bd31cfeab7a. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:59:27.593423 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48eeb721a60c4be888041bd31cfeab7a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.593834 10486 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:27.594612 7182 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:27.597394 6643 catalog_manager.cc:5484] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 4 to 5, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:27.597571 6783 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:27.618865 8509 raft_consensus.cc:1203] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:59:27.626161 9848 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.627070 7181 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.633810 6636 catalog_manager.cc:5484] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.639019 7050 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.651151 7181 consensus_queue.cc:237] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:27.653674 7050 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:59:27.653664 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6ac26fa6de44f7ca68e6c0b6b959db7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.654232 9662 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:27.654719 6783 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:27.655177 9662 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:27.659117 9767 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:27.659343 7050 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
W20240502 01:59:27.660034 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6ac26fa6de44f7ca68e6c0b6b959db7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.662124 6637 catalog_manager.cc:5484] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 4 to 5, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:27.662207 6783 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:27.678939 7050 consensus_queue.cc:237] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:27.680639 6917 raft_consensus.cc:1261] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.681062 6783 raft_consensus.cc:1261] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.681071 10511 consensus_queue.cc:1035] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:59:27.681716 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0e05f96dbc542eb999c16dcc0620666. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.681738 10511 consensus_queue.cc:1035] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 01:59:27.684545 10511 raft_consensus.cc:2932] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.686780 6917 raft_consensus.cc:2932] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.688705 6636 catalog_manager.cc:5484] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.690167 6783 raft_consensus.cc:2932] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:27.695017 8903 raft_consensus.cc:1203] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:27.705708 6783 consensus_queue.cc:237] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:27.708052 7181 raft_consensus.cc:1261] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:59:27.708285 6721 consensus_peers.cc:485] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8c237c50773494abf8813494be0f184. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.708580 10395 consensus_queue.cc:1035] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.711879 7050 raft_consensus.cc:1261] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.712915 10394 consensus_queue.cc:1035] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.713032 6916 consensus_queue.cc:237] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:27.714560 6783 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.714653 7181 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:59:27.714797 6849 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44c55b199a24432e902cce9121fc0780. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.715066 9882 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.715399 10690 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.717533 10394 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.720413 7182 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.721309 7050 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.723986 6637 catalog_manager.cc:5484] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index -1 to 4, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:27.730981 6783 consensus_queue.cc:237] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:27.732748 6916 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.732779 7050 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.733284 10591 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.733640 10591 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.734747 10404 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
W20240502 01:59:27.735543 6718 consensus_peers.cc:485] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0359dfbfd11746f1bf62a465f14634ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.737485 7181 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:27.738653 6783 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:27.740669 6635 catalog_manager.cc:5484] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:27.747085 8641 raft_consensus.cc:1203] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:27.768390 10395 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:27.768738 7050 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:27.770784 6916 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:27.773641 6643 catalog_manager.cc:5484] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:27.779078 10170 ts_tablet_manager.cc:902] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:27.779527 10170 tablet_copy_client.cc:308] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:27.781047 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet a4fb9af11abc49d496b5c3d8048c22c8 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 ({username='slave'} at 127.27.111.199:58595)
I20240502 01:59:27.781153 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet a4fb9af11abc49d496b5c3d8048c22c8 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 at {username='slave'} at 127.27.111.199:58595: session id = 4e6e76b381af4cdc90bd3e9e6e8b5cf2-a4fb9af11abc49d496b5c3d8048c22c8
I20240502 01:59:27.782184 6805 tablet_copy_source_session.cc:205] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:27.783808 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4fb9af11abc49d496b5c3d8048c22c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:27.785894 10170 tablet_copy_client.cc:783] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:27.786046 10170 tablet_copy_client.cc:647] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:27.786950 8509 raft_consensus.cc:1203] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:27.792238 10170 tablet_copy_client.cc:523] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:27.794296 10170 tablet_bootstrap.cc:492] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap starting.
I20240502 01:59:27.803586 6783 consensus_queue.cc:237] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:27.805696 7182 raft_consensus.cc:1261] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.806461 7050 raft_consensus.cc:1261] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:27.807001 10395 consensus_queue.cc:1035] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:27.807958 10395 consensus_queue.cc:1035] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:59:27.811672 6721 consensus_peers.cc:485] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e44267bd266407f9b72c66e3d3579bb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.813757 7049 consensus_queue.cc:237] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
W20240502 01:59:27.816395 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48eeb721a60c4be888041bd31cfeab7a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.817257 7181 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 01:59:27.817637 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48eeb721a60c4be888041bd31cfeab7a. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:59:27.817951 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48eeb721a60c4be888041bd31cfeab7a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:27.818065 9843 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:27.819809 6783 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:27.820230 9843 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:27.836143 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.840721 10395 raft_consensus.cc:2932] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:27.843137 7182 raft_consensus.cc:2932] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:27.843415 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.843580 7050 raft_consensus.cc:2932] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:27.846236 6637 catalog_manager.cc:5484] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:27.849642 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.852685 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:27.862601 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:27.868765 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:27.887879 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.891243 8757 raft_consensus.cc:1203] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:27.898830 10170 tablet_bootstrap.cc:492] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:27.899163 10170 tablet_bootstrap.cc:492] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap complete.
I20240502 01:59:27.899375 10170 ts_tablet_manager.cc:1367] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent bootstrapping tablet: real 0.105s user 0.029s sys 0.001s
I20240502 01:59:27.901208 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:27.901481 10170 raft_consensus.cc:348] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:27.901775 10170 raft_consensus.cc:729] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, State: Initialized, Role: FOLLOWER
I20240502 01:59:27.901960 10170 consensus_queue.cc:260] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:27.902878 10170 ts_tablet_manager.cc:1398] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:59:27.903381 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-a4fb9af11abc49d496b5c3d8048c22c8 received from {username='slave'} at 127.27.111.199:58595
I20240502 01:59:27.903463 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-a4fb9af11abc49d496b5c3d8048c22c8 on tablet a4fb9af11abc49d496b5c3d8048c22c8 with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:27.917800 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.924192 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.930047 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:27.934675 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.939389 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:27.953985 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.955094 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:27.988833 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:28.019788 10860 ts_tablet_manager.cc:902] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:28.022854 10860 tablet_copy_client.cc:308] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:28.024245 10877 ts_tablet_manager.cc:902] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:28.024818 10877 tablet_copy_client.cc:308] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:28.025576 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 47a276e9780e448a9b3c6eab67925514 from peer 4f53d80e39be410d92d4766692aff1c6 ({username='slave'} at 127.27.111.198:57069)
I20240502 01:59:28.025694 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 47a276e9780e448a9b3c6eab67925514 from peer 4f53d80e39be410d92d4766692aff1c6 at {username='slave'} at 127.27.111.198:57069: session id = 4f53d80e39be410d92d4766692aff1c6-47a276e9780e448a9b3c6eab67925514
I20240502 01:59:28.026829 7070 tablet_copy_source_session.cc:205] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.028151 10877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a276e9780e448a9b3c6eab67925514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.031100 10877 tablet_copy_client.cc:783] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.031327 10877 tablet_copy_client.cc:647] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.038969 10877 tablet_copy_client.cc:523] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.041641 10877 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap starting.
I20240502 01:59:28.050719 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet 91db5c1b3d2f4d22b5882a61ea0395c7 from peer 4f53d80e39be410d92d4766692aff1c6 ({username='slave'} at 127.27.111.198:60983)
I20240502 01:59:28.050864 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet 91db5c1b3d2f4d22b5882a61ea0395c7 from peer 4f53d80e39be410d92d4766692aff1c6 at {username='slave'} at 127.27.111.198:60983: session id = 4f53d80e39be410d92d4766692aff1c6-91db5c1b3d2f4d22b5882a61ea0395c7
I20240502 01:59:28.051906 6805 tablet_copy_source_session.cc:205] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.052789 10860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91db5c1b3d2f4d22b5882a61ea0395c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.055052 10860 tablet_copy_client.cc:783] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.055225 10860 tablet_copy_client.cc:647] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.056816 10860 tablet_copy_client.cc:523] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.058214 10860 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap starting.
I20240502 01:59:28.062249 7181 consensus_queue.cc:237] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:28.063818 6783 raft_consensus.cc:1261] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:28.064188 6916 raft_consensus.cc:1261] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:28.064446 10620 consensus_queue.cc:1035] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:59:28.065126 7117 consensus_peers.cc:485] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c54b912f802461c8cac092ac3222c87. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:28.065344 9662 consensus_queue.cc:1035] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:28.072712 10620 raft_consensus.cc:2932] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:28.073056 6783 raft_consensus.cc:2932] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:28.075352 6916 raft_consensus.cc:2932] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:28.078655 6635 catalog_manager.cc:5484] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:28.091703 10910 ts_tablet_manager.cc:902] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:28.092281 10910 tablet_copy_client.cc:308] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:28.093189 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet a66b932d884e4df8a78841103d88ded0 from peer 4f53d80e39be410d92d4766692aff1c6 ({username='slave'} at 127.27.111.198:60983)
I20240502 01:59:28.093291 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet a66b932d884e4df8a78841103d88ded0 from peer 4f53d80e39be410d92d4766692aff1c6 at {username='slave'} at 127.27.111.198:60983: session id = 4f53d80e39be410d92d4766692aff1c6-a66b932d884e4df8a78841103d88ded0
I20240502 01:59:28.094261 6805 tablet_copy_source_session.cc:205] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.095500 10910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a66b932d884e4df8a78841103d88ded0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.098052 10910 tablet_copy_client.cc:783] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.098268 10910 tablet_copy_client.cc:647] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.099736 10910 tablet_copy_client.cc:523] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.101083 10910 tablet_bootstrap.cc:492] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap starting.
I20240502 01:59:28.105685 10877 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:28.106091 10877 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap complete.
I20240502 01:59:28.106323 10877 ts_tablet_manager.cc:1367] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6: Time spent bootstrapping tablet: real 0.065s user 0.027s sys 0.000s
I20240502 01:59:28.108150 10877 raft_consensus.cc:348] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:28.108371 10877 raft_consensus.cc:729] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.108585 10877 consensus_queue.cc:260] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:28.109589 10877 ts_tablet_manager.cc:1398] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:28.110065 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 4f53d80e39be410d92d4766692aff1c6-47a276e9780e448a9b3c6eab67925514 received from {username='slave'} at 127.27.111.198:57069
I20240502 01:59:28.110149 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 4f53d80e39be410d92d4766692aff1c6-47a276e9780e448a9b3c6eab67925514 on tablet 47a276e9780e448a9b3c6eab67925514 with peer 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:28.116528 10860 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:28.116976 10860 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap complete.
I20240502 01:59:28.117214 10860 ts_tablet_manager.cc:1367] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6: Time spent bootstrapping tablet: real 0.059s user 0.029s sys 0.000s
I20240502 01:59:28.117741 10860 raft_consensus.cc:348] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:28.117935 10860 raft_consensus.cc:729] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.119704 10860 consensus_queue.cc:260] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:28.120132 10860 ts_tablet_manager.cc:1398] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:28.120684 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 4f53d80e39be410d92d4766692aff1c6-91db5c1b3d2f4d22b5882a61ea0395c7 received from {username='slave'} at 127.27.111.198:60983
I20240502 01:59:28.120772 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 4f53d80e39be410d92d4766692aff1c6-91db5c1b3d2f4d22b5882a61ea0395c7 on tablet 91db5c1b3d2f4d22b5882a61ea0395c7 with peer 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:28.127532 10949 ts_tablet_manager.cc:902] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:28.128340 10949 tablet_copy_client.cc:308] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 01:59:28.129792 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet 44c55b199a24432e902cce9121fc0780 from peer 070558c278e0457b9cc070ac0d662b5d ({username='slave'} at 127.27.111.200:54661)
I20240502 01:59:28.129920 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet 44c55b199a24432e902cce9121fc0780 from peer 070558c278e0457b9cc070ac0d662b5d at {username='slave'} at 127.27.111.200:54661: session id = 070558c278e0457b9cc070ac0d662b5d-44c55b199a24432e902cce9121fc0780
I20240502 01:59:28.130937 6937 tablet_copy_source_session.cc:205] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.133968 10910 tablet_bootstrap.cc:492] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:28.134413 10910 tablet_bootstrap.cc:492] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap complete.
I20240502 01:59:28.134711 10910 ts_tablet_manager.cc:1367] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6: Time spent bootstrapping tablet: real 0.034s user 0.030s sys 0.000s
I20240502 01:59:28.135188 10949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c55b199a24432e902cce9121fc0780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.135254 10910 raft_consensus.cc:348] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:28.135365 10910 raft_consensus.cc:729] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.135486 10910 consensus_queue.cc:260] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:28.136312 10910 ts_tablet_manager.cc:1398] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:28.137655 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 4f53d80e39be410d92d4766692aff1c6-a66b932d884e4df8a78841103d88ded0 received from {username='slave'} at 127.27.111.198:60983
I20240502 01:59:28.137746 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 4f53d80e39be410d92d4766692aff1c6-a66b932d884e4df8a78841103d88ded0 on tablet a66b932d884e4df8a78841103d88ded0 with peer 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:28.139482 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.153214 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.154594 10949 tablet_copy_client.cc:783] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.154821 10949 tablet_copy_client.cc:647] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.155537 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.173488 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:28.180289 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.180955 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:28.184563 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:28.185562 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.193233 10170 ts_tablet_manager.cc:902] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:28.193720 10170 tablet_copy_client.cc:308] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:36569
I20240502 01:59:28.194254 7201 tablet_copy_service.cc:140] P 86bc8001c3464b929cd5c873861baa0c: Received BeginTabletCopySession request for tablet e6ac26fa6de44f7ca68e6c0b6b959db7 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 ({username='slave'} at 127.27.111.199:47693)
I20240502 01:59:28.194379 7201 tablet_copy_service.cc:161] P 86bc8001c3464b929cd5c873861baa0c: Beginning new tablet copy session on tablet e6ac26fa6de44f7ca68e6c0b6b959db7 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 at {username='slave'} at 127.27.111.199:47693: session id = 4e6e76b381af4cdc90bd3e9e6e8b5cf2-e6ac26fa6de44f7ca68e6c0b6b959db7
I20240502 01:59:28.195140 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:28.195417 7201 tablet_copy_source_session.cc:205] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.196372 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ac26fa6de44f7ca68e6c0b6b959db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.197656 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:28.198750 10170 tablet_copy_client.cc:783] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.199142 10170 tablet_copy_client.cc:647] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.199846 10910 ts_tablet_manager.cc:902] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:28.200323 10910 tablet_copy_client.cc:308] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:28.200733 10170 tablet_copy_client.cc:523] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.200874 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 48eeb721a60c4be888041bd31cfeab7a from peer 4f53d80e39be410d92d4766692aff1c6 ({username='slave'} at 127.27.111.198:57069)
I20240502 01:59:28.200990 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 48eeb721a60c4be888041bd31cfeab7a from peer 4f53d80e39be410d92d4766692aff1c6 at {username='slave'} at 127.27.111.198:57069: session id = 4f53d80e39be410d92d4766692aff1c6-48eeb721a60c4be888041bd31cfeab7a
I20240502 01:59:28.202076 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:28.202091 7070 tablet_copy_source_session.cc:205] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.202437 10170 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap starting.
I20240502 01:59:28.203050 10910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48eeb721a60c4be888041bd31cfeab7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.205214 10910 tablet_copy_client.cc:783] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.205365 10910 tablet_copy_client.cc:647] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.212872 7181 consensus_queue.cc:237] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:28.217621 6916 raft_consensus.cc:1261] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:28.217705 10986 ts_tablet_manager.cc:902] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:28.218287 9767 consensus_queue.cc:1035] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:28.218497 7050 raft_consensus.cc:1261] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:28.218695 10986 tablet_copy_client.cc:308] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:36569
W20240502 01:59:28.219278 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7fa358140eb742b587a4d6a203738b4f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:28.219754 9767 consensus_queue.cc:1035] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:28.221158 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.230206 10170 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:28.230615 10170 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap complete.
I20240502 01:59:28.230829 10170 ts_tablet_manager.cc:1367] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent bootstrapping tablet: real 0.028s user 0.026s sys 0.001s
I20240502 01:59:28.231302 10170 raft_consensus.cc:348] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:28.231437 10170 raft_consensus.cc:729] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.231598 10170 consensus_queue.cc:260] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:28.231909 10170 ts_tablet_manager.cc:1398] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:28.232570 7201 tablet_copy_service.cc:342] P 86bc8001c3464b929cd5c873861baa0c: Request end of tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-e6ac26fa6de44f7ca68e6c0b6b959db7 received from {username='slave'} at 127.27.111.199:47693
I20240502 01:59:28.232602 10949 tablet_copy_client.cc:523] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.232664 7201 tablet_copy_service.cc:433] P 86bc8001c3464b929cd5c873861baa0c: ending tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-e6ac26fa6de44f7ca68e6c0b6b959db7 on tablet e6ac26fa6de44f7ca68e6c0b6b959db7 with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:28.234323 10994 ts_tablet_manager.cc:902] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:28.234369 10949 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap starting.
I20240502 01:59:28.234982 10994 tablet_copy_client.cc:308] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:28.235239 10860 ts_tablet_manager.cc:902] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:28.235482 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet 0359dfbfd11746f1bf62a465f14634ec from peer 070558c278e0457b9cc070ac0d662b5d ({username='slave'} at 127.27.111.200:36335)
I20240502 01:59:28.235613 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet 0359dfbfd11746f1bf62a465f14634ec from peer 070558c278e0457b9cc070ac0d662b5d at {username='slave'} at 127.27.111.200:36335: session id = 070558c278e0457b9cc070ac0d662b5d-0359dfbfd11746f1bf62a465f14634ec
I20240502 01:59:28.235806 10860 tablet_copy_client.cc:308] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:28.236570 6805 tablet_copy_source_session.cc:205] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.237449 10994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0359dfbfd11746f1bf62a465f14634ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.238034 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet c8c237c50773494abf8813494be0f184 from peer 4f53d80e39be410d92d4766692aff1c6 ({username='slave'} at 127.27.111.198:60983)
I20240502 01:59:28.238147 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet c8c237c50773494abf8813494be0f184 from peer 4f53d80e39be410d92d4766692aff1c6 at {username='slave'} at 127.27.111.198:60983: session id = 4f53d80e39be410d92d4766692aff1c6-c8c237c50773494abf8813494be0f184
I20240502 01:59:28.239030 6805 tablet_copy_source_session.cc:205] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.239948 10860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c237c50773494abf8813494be0f184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.240204 10994 tablet_copy_client.cc:783] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.240346 10994 tablet_copy_client.cc:647] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.241467 10994 tablet_copy_client.cc:523] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.241991 10860 tablet_copy_client.cc:783] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.242132 10860 tablet_copy_client.cc:647] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.243034 10994 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d: Bootstrap starting.
I20240502 01:59:28.250445 10860 tablet_copy_client.cc:523] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.252241 10860 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap starting.
I20240502 01:59:28.261493 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.261595 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.264374 8757 raft_consensus.cc:1203] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:28.267038 10170 ts_tablet_manager.cc:902] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:28.268388 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:28.271374 7201 tablet_copy_service.cc:140] P 86bc8001c3464b929cd5c873861baa0c: Received BeginTabletCopySession request for tablet e6ac26fa6de44f7ca68e6c0b6b959db7 from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:59373)
I20240502 01:59:28.271509 7201 tablet_copy_service.cc:161] P 86bc8001c3464b929cd5c873861baa0c: Beginning new tablet copy session on tablet e6ac26fa6de44f7ca68e6c0b6b959db7 from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:59373: session id = 954cd49326bf405abd81182d75dc783c-e6ac26fa6de44f7ca68e6c0b6b959db7
I20240502 01:59:28.272547 7201 tablet_copy_source_session.cc:205] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.273448 10986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ac26fa6de44f7ca68e6c0b6b959db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.275766 10986 tablet_copy_client.cc:783] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.275938 10986 tablet_copy_client.cc:647] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.276304 10170 tablet_copy_client.cc:308] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:28.282436 10986 tablet_copy_client.cc:523] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.283963 10986 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 01:59:28.295692 9767 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:28.296916 10910 tablet_copy_client.cc:523] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.296986 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:28.298522 10910 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6: Bootstrap starting.
I20240502 01:59:28.303381 6636 catalog_manager.cc:5484] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:28.312208 6916 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:28.312480 7050 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:28.350868 10994 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:28.351167 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet e0e05f96dbc542eb999c16dcc0620666 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 ({username='slave'} at 127.27.111.199:49969)
I20240502 01:59:28.351281 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet e0e05f96dbc542eb999c16dcc0620666 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 at {username='slave'} at 127.27.111.199:49969: session id = 4e6e76b381af4cdc90bd3e9e6e8b5cf2-e0e05f96dbc542eb999c16dcc0620666
I20240502 01:59:28.352450 7070 tablet_copy_source_session.cc:205] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.352842 10994 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d: Bootstrap complete.
I20240502 01:59:28.353073 10994 ts_tablet_manager.cc:1367] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d: Time spent bootstrapping tablet: real 0.110s user 0.027s sys 0.000s
I20240502 01:59:28.353559 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e05f96dbc542eb999c16dcc0620666. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.353561 10994 raft_consensus.cc:348] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:28.353682 10994 raft_consensus.cc:729] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.355942 10170 tablet_copy_client.cc:783] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.356098 10170 tablet_copy_client.cc:647] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.357429 10170 tablet_copy_client.cc:523] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.359598 10994 consensus_queue.cc:260] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:28.360057 10994 ts_tablet_manager.cc:1398] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:59:28.360672 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 070558c278e0457b9cc070ac0d662b5d-0359dfbfd11746f1bf62a465f14634ec received from {username='slave'} at 127.27.111.200:36335
I20240502 01:59:28.360754 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 070558c278e0457b9cc070ac0d662b5d-0359dfbfd11746f1bf62a465f14634ec on tablet 0359dfbfd11746f1bf62a465f14634ec with peer 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:28.361518 10170 tablet_bootstrap.cc:492] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap starting.
I20240502 01:59:28.364223 10986 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:28.364565 10986 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
I20240502 01:59:28.364773 10986 ts_tablet_manager.cc:1367] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.081s user 0.031s sys 0.000s
I20240502 01:59:28.365348 10986 raft_consensus.cc:348] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:28.365520 10986 raft_consensus.cc:729] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.365695 10986 consensus_queue.cc:260] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:28.366137 10986 ts_tablet_manager.cc:1398] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:28.370946 7201 tablet_copy_service.cc:342] P 86bc8001c3464b929cd5c873861baa0c: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-e6ac26fa6de44f7ca68e6c0b6b959db7 received from {username='slave'} at 127.27.111.197:59373
I20240502 01:59:28.371050 7201 tablet_copy_service.cc:433] P 86bc8001c3464b929cd5c873861baa0c: ending tablet copy session 954cd49326bf405abd81182d75dc783c-e6ac26fa6de44f7ca68e6c0b6b959db7 on tablet e6ac26fa6de44f7ca68e6c0b6b959db7 with peer 954cd49326bf405abd81182d75dc783c
I20240502 01:59:28.390168 10170 tablet_bootstrap.cc:492] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:28.390530 10170 tablet_bootstrap.cc:492] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap complete.
I20240502 01:59:28.390733 10170 ts_tablet_manager.cc:1367] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.011s
I20240502 01:59:28.391314 10170 raft_consensus.cc:348] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:28.391481 10170 raft_consensus.cc:729] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.391654 10170 consensus_queue.cc:260] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:28.391963 10170 ts_tablet_manager.cc:1398] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:28.392494 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-e0e05f96dbc542eb999c16dcc0620666 received from {username='slave'} at 127.27.111.199:49969
I20240502 01:59:28.392570 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-e0e05f96dbc542eb999c16dcc0620666 on tablet e0e05f96dbc542eb999c16dcc0620666 with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:28.395591 10170 ts_tablet_manager.cc:902] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:28.396173 10170 tablet_copy_client.cc:308] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:28.396525 10910 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:59:28.396589 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 48eeb721a60c4be888041bd31cfeab7a from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 ({username='slave'} at 127.27.111.199:49969)
I20240502 01:59:28.396692 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 48eeb721a60c4be888041bd31cfeab7a from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 at {username='slave'} at 127.27.111.199:49969: session id = 4e6e76b381af4cdc90bd3e9e6e8b5cf2-48eeb721a60c4be888041bd31cfeab7a
I20240502 01:59:28.396895 10910 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6: Bootstrap complete.
I20240502 01:59:28.397058 10910 ts_tablet_manager.cc:1367] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6: Time spent bootstrapping tablet: real 0.099s user 0.022s sys 0.004s
I20240502 01:59:28.397480 10910 raft_consensus.cc:348] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:28.397588 7070 tablet_copy_source_session.cc:205] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.397960 10910 raft_consensus.cc:729] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.398066 10910 consensus_queue.cc:260] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:28.398483 10170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48eeb721a60c4be888041bd31cfeab7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.398988 10910 ts_tablet_manager.cc:1398] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:28.399492 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 4f53d80e39be410d92d4766692aff1c6-48eeb721a60c4be888041bd31cfeab7a received from {username='slave'} at 127.27.111.198:57069
I20240502 01:59:28.399556 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 4f53d80e39be410d92d4766692aff1c6-48eeb721a60c4be888041bd31cfeab7a on tablet 48eeb721a60c4be888041bd31cfeab7a with peer 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:28.401154 10170 tablet_copy_client.cc:783] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.401329 10170 tablet_copy_client.cc:647] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.402298 10170 tablet_copy_client.cc:523] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.403630 10170 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap starting.
I20240502 01:59:28.409652 10949 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:28.410073 10949 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap complete.
I20240502 01:59:28.410291 10949 ts_tablet_manager.cc:1367] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d: Time spent bootstrapping tablet: real 0.176s user 0.030s sys 0.000s
I20240502 01:59:28.410820 10949 raft_consensus.cc:348] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:28.411127 10949 raft_consensus.cc:729] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.411322 10949 consensus_queue.cc:260] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:28.411733 10949 ts_tablet_manager.cc:1398] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:28.415460 10986 ts_tablet_manager.cc:902] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:28.415930 10986 tablet_copy_client.cc:308] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:28.417151 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 48eeb721a60c4be888041bd31cfeab7a from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:40005)
I20240502 01:59:28.417249 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 48eeb721a60c4be888041bd31cfeab7a from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:40005: session id = 954cd49326bf405abd81182d75dc783c-48eeb721a60c4be888041bd31cfeab7a
I20240502 01:59:28.418177 7070 tablet_copy_source_session.cc:205] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.419500 10986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48eeb721a60c4be888041bd31cfeab7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.420954 11052 ts_tablet_manager.cc:902] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:28.421469 11052 tablet_copy_client.cc:308] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:28.421963 10986 tablet_copy_client.cc:783] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.423833 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 070558c278e0457b9cc070ac0d662b5d-44c55b199a24432e902cce9121fc0780 received from {username='slave'} at 127.27.111.200:54661
I20240502 01:59:28.423918 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 070558c278e0457b9cc070ac0d662b5d-44c55b199a24432e902cce9121fc0780 on tablet 44c55b199a24432e902cce9121fc0780 with peer 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:28.428277 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet 4e44267bd266407f9b72c66e3d3579bb from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:33267)
I20240502 01:59:28.428380 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet 4e44267bd266407f9b72c66e3d3579bb from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:33267: session id = 954cd49326bf405abd81182d75dc783c-4e44267bd266407f9b72c66e3d3579bb
I20240502 01:59:28.429329 6805 tablet_copy_source_session.cc:205] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.430114 10986 tablet_copy_client.cc:647] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.431473 10986 tablet_copy_client.cc:523] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.433266 10170 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:59:28.433658 10170 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap complete.
I20240502 01:59:28.433898 10170 ts_tablet_manager.cc:1367] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.002s
I20240502 01:59:28.434547 10170 raft_consensus.cc:348] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:28.435120 10170 raft_consensus.cc:729] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.435240 10170 consensus_queue.cc:260] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:28.435557 10170 ts_tablet_manager.cc:1398] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:28.436117 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-48eeb721a60c4be888041bd31cfeab7a received from {username='slave'} at 127.27.111.199:49969
I20240502 01:59:28.436200 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-48eeb721a60c4be888041bd31cfeab7a on tablet 48eeb721a60c4be888041bd31cfeab7a with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:28.438843 10986 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 01:59:28.442770 11052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e44267bd266407f9b72c66e3d3579bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.445153 11052 tablet_copy_client.cc:783] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.445385 11052 tablet_copy_client.cc:647] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.446878 11052 tablet_copy_client.cc:523] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.448534 10860 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:28.448928 10860 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap complete.
I20240502 01:59:28.449132 10860 ts_tablet_manager.cc:1367] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6: Time spent bootstrapping tablet: real 0.197s user 0.030s sys 0.000s
I20240502 01:59:28.449828 10860 raft_consensus.cc:348] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:28.451110 10860 raft_consensus.cc:729] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.451285 10860 consensus_queue.cc:260] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:28.452288 10860 ts_tablet_manager.cc:1398] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:59:28.453429 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 4f53d80e39be410d92d4766692aff1c6-c8c237c50773494abf8813494be0f184 received from {username='slave'} at 127.27.111.198:60983
I20240502 01:59:28.453512 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 4f53d80e39be410d92d4766692aff1c6-c8c237c50773494abf8813494be0f184 on tablet c8c237c50773494abf8813494be0f184 with peer 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:28.463114 11052 tablet_bootstrap.cc:492] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 01:59:28.466773 11085 ts_tablet_manager.cc:902] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:28.467417 11085 tablet_copy_client.cc:308] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:36569
I20240502 01:59:28.467890 7201 tablet_copy_service.cc:140] P 86bc8001c3464b929cd5c873861baa0c: Received BeginTabletCopySession request for tablet 1c54b912f802461c8cac092ac3222c87 from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:59373)
I20240502 01:59:28.468024 7201 tablet_copy_service.cc:161] P 86bc8001c3464b929cd5c873861baa0c: Beginning new tablet copy session on tablet 1c54b912f802461c8cac092ac3222c87 from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:59373: session id = 954cd49326bf405abd81182d75dc783c-1c54b912f802461c8cac092ac3222c87
I20240502 01:59:28.469072 7201 tablet_copy_source_session.cc:205] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.470029 11085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c54b912f802461c8cac092ac3222c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.473489 11085 tablet_copy_client.cc:783] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.473729 11085 tablet_copy_client.cc:647] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.482285 7050 consensus_queue.cc:237] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
W20240502 01:59:28.484648 6985 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b69d6e907b3b400f93721981732c006b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:28.484709 6916 raft_consensus.cc:1261] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:28.485291 6783 raft_consensus.cc:1261] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:28.485272 10486 consensus_queue.cc:1035] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:28.486848 10486 consensus_queue.cc:1035] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:28.492902 10986 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:59:28.493273 10986 tablet_bootstrap.cc:492] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
I20240502 01:59:28.493438 10986 ts_tablet_manager.cc:1367] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.059s user 0.028s sys 0.000s
I20240502 01:59:28.493963 10986 raft_consensus.cc:348] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:28.494504 10986 raft_consensus.cc:729] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.494619 10986 consensus_queue.cc:260] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:28.495011 10986 ts_tablet_manager.cc:1398] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:59:28.497104 11085 tablet_copy_client.cc:523] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.497665 9843 raft_consensus.cc:2932] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:28.498118 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-48eeb721a60c4be888041bd31cfeab7a received from {username='slave'} at 127.27.111.197:40005
I20240502 01:59:28.498188 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 954cd49326bf405abd81182d75dc783c-48eeb721a60c4be888041bd31cfeab7a on tablet 48eeb721a60c4be888041bd31cfeab7a with peer 954cd49326bf405abd81182d75dc783c
I20240502 01:59:28.499084 11085 tablet_bootstrap.cc:492] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 01:59:28.499756 6916 raft_consensus.cc:2932] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:28.500838 6783 raft_consensus.cc:2932] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:28.507977 6641 catalog_manager.cc:5484] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:28.525549 6916 consensus_queue.cc:237] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:28.525768 11052 tablet_bootstrap.cc:492] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:28.528381 7050 raft_consensus.cc:1261] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:28.528506 11052 tablet_bootstrap.cc:492] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
W20240502 01:59:28.528643 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcc5daef41bc44c4855f2bcb44e1c6d5. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:28.528700 11052 ts_tablet_manager.cc:1367] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.066s user 0.022s sys 0.007s
I20240502 01:59:28.528728 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.529047 7181 raft_consensus.cc:1261] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:28.529448 11139 consensus_queue.cc:1035] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:28.530887 11139 consensus_queue.cc:1035] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:28.535646 8641 raft_consensus.cc:1203] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:28.538440 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.547024 11052 raft_consensus.cc:348] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:28.547195 11052 raft_consensus.cc:729] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.547174 8903 raft_consensus.cc:1203] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:28.549839 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.550697 11052 consensus_queue.cc:260] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:28.552546 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:28.558017 8641 raft_consensus.cc:1203] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:28.563230 11085 tablet_bootstrap.cc:492] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:28.563592 11085 tablet_bootstrap.cc:492] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
I20240502 01:59:28.563863 11085 ts_tablet_manager.cc:1367] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.065s user 0.025s sys 0.004s
I20240502 01:59:28.564350 11085 raft_consensus.cc:348] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:28.564553 11085 raft_consensus.cc:729] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.564754 11085 consensus_queue.cc:260] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:28.565356 11085 ts_tablet_manager.cc:1398] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:28.565809 7201 tablet_copy_service.cc:342] P 86bc8001c3464b929cd5c873861baa0c: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-1c54b912f802461c8cac092ac3222c87 received from {username='slave'} at 127.27.111.197:59373
I20240502 01:59:28.565898 7201 tablet_copy_service.cc:433] P 86bc8001c3464b929cd5c873861baa0c: ending tablet copy session 954cd49326bf405abd81182d75dc783c-1c54b912f802461c8cac092ac3222c87 on tablet 1c54b912f802461c8cac092ac3222c87 with peer 954cd49326bf405abd81182d75dc783c
I20240502 01:59:28.566190 11052 ts_tablet_manager.cc:1398] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20240502 01:59:28.567605 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-4e44267bd266407f9b72c66e3d3579bb received from {username='slave'} at 127.27.111.197:33267
I20240502 01:59:28.567699 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 954cd49326bf405abd81182d75dc783c-4e44267bd266407f9b72c66e3d3579bb on tablet 4e44267bd266407f9b72c66e3d3579bb with peer 954cd49326bf405abd81182d75dc783c
I20240502 01:59:28.572824 7182 consensus_queue.cc:237] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:28.574460 8509 raft_consensus.cc:1261] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:28.574621 6916 raft_consensus.cc:1261] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:59:28.574987 7114 consensus_peers.cc:485] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c54b912f802461c8cac092ac3222c87. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:28.575148 9658 consensus_queue.cc:1035] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:28.575479 8963 consensus_queue.cc:1035] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:28.575933 6783 raft_consensus.cc:1261] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:28.576375 9658 consensus_queue.cc:1035] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:28.579133 9658 raft_consensus.cc:2932] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:28.579600 8509 raft_consensus.cc:2932] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:28.581451 6916 raft_consensus.cc:2932] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:28.582595 6641 catalog_manager.cc:5484] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 4 to 5, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:28.583818 6783 raft_consensus.cc:2932] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:28.591629 8641 raft_consensus.cc:1203] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:28.592417 10723 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:28.594774 7181 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:28.595556 7050 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:28.596200 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:28.597128 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:28.597621 6636 catalog_manager.cc:5484] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index -1 to 4, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:28.597954 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:28.603137 8641 raft_consensus.cc:1203] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:59:28.605244 11109 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:28.605607 8641 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:28.608408 6785 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:28.611491 6635 catalog_manager.cc:5484] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:28.615803 8508 raft_consensus.cc:1203] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:59:28.618660 7181 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:28.623521 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.628202 8760 raft_consensus.cc:1203] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:59:28.662496 8903 raft_consensus.cc:1203] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:28.695960 7050 consensus_queue.cc:237] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:28.698002 6916 raft_consensus.cc:1261] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:59:28.698170 6985 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b69d6e907b3b400f93721981732c006b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:28.698462 11167 consensus_queue.cc:1035] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:28.698863 6785 raft_consensus.cc:1261] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:28.699260 11167 consensus_queue.cc:1035] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:59:28.700141 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b69d6e907b3b400f93721981732c006b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:28.701057 11167 raft_consensus.cc:2932] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:28.704349 6785 raft_consensus.cc:2932] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:28.707755 6916 raft_consensus.cc:2932] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:28.707893 6641 catalog_manager.cc:5484] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 4 to 5, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:28.750604 8760 raft_consensus.cc:1203] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:28.763330 7050 consensus_queue.cc:237] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:28.765385 7181 raft_consensus.cc:1261] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:28.766309 6916 raft_consensus.cc:1261] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:59:28.767271 10486 consensus_queue.cc:1035] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:59:28.767835 6985 consensus_peers.cc:485] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4d429fe60e74108bc0c9ca8509365f2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:28.769753 10486 consensus_queue.cc:1035] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:59:28.777330 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.783392 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.784793 10486 raft_consensus.cc:2932] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:28.787277 7181 raft_consensus.cc:2932] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:28.790469 6916 raft_consensus.cc:2932] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:28.795760 6642 catalog_manager.cc:5484] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index -1 to 4, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:28.799989 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:28.800863 10949 ts_tablet_manager.cc:902] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:28.801309 10949 tablet_copy_client.cc:308] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:36569
I20240502 01:59:28.801815 7201 tablet_copy_service.cc:140] P 86bc8001c3464b929cd5c873861baa0c: Received BeginTabletCopySession request for tablet 7fa358140eb742b587a4d6a203738b4f from peer 070558c278e0457b9cc070ac0d662b5d ({username='slave'} at 127.27.111.200:56521)
I20240502 01:59:28.801916 7201 tablet_copy_service.cc:161] P 86bc8001c3464b929cd5c873861baa0c: Beginning new tablet copy session on tablet 7fa358140eb742b587a4d6a203738b4f from peer 070558c278e0457b9cc070ac0d662b5d at {username='slave'} at 127.27.111.200:56521: session id = 070558c278e0457b9cc070ac0d662b5d-7fa358140eb742b587a4d6a203738b4f
I20240502 01:59:28.802865 7201 tablet_copy_source_session.cc:205] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.803750 10949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fa358140eb742b587a4d6a203738b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.806116 10949 tablet_copy_client.cc:783] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.806265 10949 tablet_copy_client.cc:647] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:28.815367 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.817077 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:28.830626 8641 raft_consensus.cc:1203] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:28.837843 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.839232 8509 raft_consensus.cc:1203] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:59:28.839411 8509 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:28.845244 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:28.857468 10949 tablet_copy_client.cc:523] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:28.859544 10949 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d: Bootstrap starting.
I20240502 01:59:28.859776 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:28.866374 8760 raft_consensus.cc:1203] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:59:28.866768 8760 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:28.880203 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.886505 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:28.900661 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.914580 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:28.920431 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.925964 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.941388 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:28.942313 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.957156 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.958808 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:28.962261 10949 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:28.978971 8509 raft_consensus.cc:1203] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:28.982620 10949 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d: Bootstrap complete.
I20240502 01:59:28.982896 10949 ts_tablet_manager.cc:1367] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d: Time spent bootstrapping tablet: real 0.123s user 0.027s sys 0.000s
I20240502 01:59:28.983397 10949 raft_consensus.cc:348] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:28.983523 10949 raft_consensus.cc:729] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Initialized, Role: FOLLOWER
I20240502 01:59:28.983642 10949 consensus_queue.cc:260] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:28.983906 10949 ts_tablet_manager.cc:1398] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:28.984436 7201 tablet_copy_service.cc:342] P 86bc8001c3464b929cd5c873861baa0c: Request end of tablet copy session 070558c278e0457b9cc070ac0d662b5d-7fa358140eb742b587a4d6a203738b4f received from {username='slave'} at 127.27.111.200:56521
I20240502 01:59:28.984526 7201 tablet_copy_service.cc:433] P 86bc8001c3464b929cd5c873861baa0c: ending tablet copy session 070558c278e0457b9cc070ac0d662b5d-7fa358140eb742b587a4d6a203738b4f on tablet 7fa358140eb742b587a4d6a203738b4f with peer 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:28.986977 10949 ts_tablet_manager.cc:902] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:28.987396 10949 tablet_copy_client.cc:308] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:36569
I20240502 01:59:28.987928 7201 tablet_copy_service.cc:140] P 86bc8001c3464b929cd5c873861baa0c: Received BeginTabletCopySession request for tablet 1c54b912f802461c8cac092ac3222c87 from peer 070558c278e0457b9cc070ac0d662b5d ({username='slave'} at 127.27.111.200:56521)
I20240502 01:59:28.988039 7201 tablet_copy_service.cc:161] P 86bc8001c3464b929cd5c873861baa0c: Beginning new tablet copy session on tablet 1c54b912f802461c8cac092ac3222c87 from peer 070558c278e0457b9cc070ac0d662b5d at {username='slave'} at 127.27.111.200:56521: session id = 070558c278e0457b9cc070ac0d662b5d-1c54b912f802461c8cac092ac3222c87
I20240502 01:59:28.989044 7201 tablet_copy_source_session.cc:205] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:28.989957 10949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c54b912f802461c8cac092ac3222c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:28.992791 10949 tablet_copy_client.cc:783] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:28.992973 10949 tablet_copy_client.cc:647] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:29.000666 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.010706 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:29.029608 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:29.030203 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.031224 10949 tablet_copy_client.cc:523] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:29.033004 10949 tablet_bootstrap.cc:492] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap starting.
I20240502 01:59:29.033691 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:29.039361 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:29.059610 11397 ts_tablet_manager.cc:902] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:29.060809 11397 tablet_copy_client.cc:308] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 01:59:29.061445 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet dcc5daef41bc44c4855f2bcb44e1c6d5 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 ({username='slave'} at 127.27.111.199:52619)
I20240502 01:59:29.061550 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet dcc5daef41bc44c4855f2bcb44e1c6d5 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 at {username='slave'} at 127.27.111.199:52619: session id = 4e6e76b381af4cdc90bd3e9e6e8b5cf2-dcc5daef41bc44c4855f2bcb44e1c6d5
I20240502 01:59:29.062608 6937 tablet_copy_source_session.cc:205] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:29.063678 11397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc5daef41bc44c4855f2bcb44e1c6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:29.089485 11397 tablet_copy_client.cc:783] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:29.089718 11397 tablet_copy_client.cc:647] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:29.102447 11397 tablet_copy_client.cc:523] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:29.104218 11397 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap starting.
I20240502 01:59:29.131035 10949 tablet_bootstrap.cc:492] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:29.131393 10949 tablet_bootstrap.cc:492] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap complete.
I20240502 01:59:29.131600 10949 ts_tablet_manager.cc:1367] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d: Time spent bootstrapping tablet: real 0.099s user 0.027s sys 0.000s
I20240502 01:59:29.132208 10949 raft_consensus.cc:348] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:29.132347 10949 raft_consensus.cc:729] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Initialized, Role: FOLLOWER
I20240502 01:59:29.132490 10949 consensus_queue.cc:260] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:29.132905 10949 ts_tablet_manager.cc:1398] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:29.133419 7201 tablet_copy_service.cc:342] P 86bc8001c3464b929cd5c873861baa0c: Request end of tablet copy session 070558c278e0457b9cc070ac0d662b5d-1c54b912f802461c8cac092ac3222c87 received from {username='slave'} at 127.27.111.200:56521
I20240502 01:59:29.133510 7201 tablet_copy_service.cc:433] P 86bc8001c3464b929cd5c873861baa0c: ending tablet copy session 070558c278e0457b9cc070ac0d662b5d-1c54b912f802461c8cac092ac3222c87 on tablet 1c54b912f802461c8cac092ac3222c87 with peer 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:29.159075 11446 ts_tablet_manager.cc:902] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:29.159689 11446 tablet_copy_client.cc:308] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:29.160161 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet b69d6e907b3b400f93721981732c006b from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:40005)
I20240502 01:59:29.160276 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet b69d6e907b3b400f93721981732c006b from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:40005: session id = 954cd49326bf405abd81182d75dc783c-b69d6e907b3b400f93721981732c006b
I20240502 01:59:29.161352 7070 tablet_copy_source_session.cc:205] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:29.163704 11446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69d6e907b3b400f93721981732c006b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:29.168299 11446 tablet_copy_client.cc:783] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:29.168586 11446 tablet_copy_client.cc:647] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:29.170140 11446 tablet_copy_client.cc:523] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:29.176071 11446 tablet_bootstrap.cc:492] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 01:59:29.177871 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.179153 11397 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:29.179474 11397 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap complete.
I20240502 01:59:29.179666 11397 ts_tablet_manager.cc:1367] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent bootstrapping tablet: real 0.076s user 0.029s sys 0.000s
I20240502 01:59:29.180202 11397 raft_consensus.cc:348] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:29.180332 11397 raft_consensus.cc:729] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, State: Initialized, Role: FOLLOWER
I20240502 01:59:29.180474 11397 consensus_queue.cc:260] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:29.181506 11397 ts_tablet_manager.cc:1398] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:59:29.182037 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-dcc5daef41bc44c4855f2bcb44e1c6d5 received from {username='slave'} at 127.27.111.199:52619
I20240502 01:59:29.182134 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-dcc5daef41bc44c4855f2bcb44e1c6d5 on tablet dcc5daef41bc44c4855f2bcb44e1c6d5 with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:29.190623 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.205796 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.208184 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:29.215474 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:29.233348 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.239487 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:29.243412 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:29.246830 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.264446 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.270996 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.271962 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:29.282366 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.286417 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:29.286872 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:29.289552 11397 ts_tablet_manager.cc:902] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:29.290022 11397 tablet_copy_client.cc:308] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:29.290555 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet b69d6e907b3b400f93721981732c006b from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 ({username='slave'} at 127.27.111.199:49969)
I20240502 01:59:29.290659 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet b69d6e907b3b400f93721981732c006b from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 at {username='slave'} at 127.27.111.199:49969: session id = 4e6e76b381af4cdc90bd3e9e6e8b5cf2-b69d6e907b3b400f93721981732c006b
I20240502 01:59:29.291667 7070 tablet_copy_source_session.cc:205] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:29.292671 11397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69d6e907b3b400f93721981732c006b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:29.294910 11397 tablet_copy_client.cc:783] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:29.295047 11397 tablet_copy_client.cc:647] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:29.321667 11446 tablet_bootstrap.cc:492] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:29.321658 11397 tablet_copy_client.cc:523] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:29.322149 11446 tablet_bootstrap.cc:492] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
I20240502 01:59:29.322448 11446 ts_tablet_manager.cc:1367] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.146s user 0.024s sys 0.003s
I20240502 01:59:29.322970 11446 raft_consensus.cc:348] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:29.323195 11446 raft_consensus.cc:729] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 01:59:29.323314 11397 tablet_bootstrap.cc:492] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap starting.
I20240502 01:59:29.323454 11446 consensus_queue.cc:260] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:29.324311 11446 ts_tablet_manager.cc:1398] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:59:29.324807 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-b69d6e907b3b400f93721981732c006b received from {username='slave'} at 127.27.111.197:40005
I20240502 01:59:29.324905 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 954cd49326bf405abd81182d75dc783c-b69d6e907b3b400f93721981732c006b on tablet b69d6e907b3b400f93721981732c006b with peer 954cd49326bf405abd81182d75dc783c
I20240502 01:59:29.325141 11526 ts_tablet_manager.cc:902] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:29.325700 11526 tablet_copy_client.cc:308] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:29.326203 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet d4d429fe60e74108bc0c9ca8509365f2 from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:40005)
I20240502 01:59:29.326306 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet d4d429fe60e74108bc0c9ca8509365f2 from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:40005: session id = 954cd49326bf405abd81182d75dc783c-d4d429fe60e74108bc0c9ca8509365f2
I20240502 01:59:29.327405 7070 tablet_copy_source_session.cc:205] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:29.330246 11526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d429fe60e74108bc0c9ca8509365f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:29.331665 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:29.334234 11526 tablet_copy_client.cc:783] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:29.334497 11526 tablet_copy_client.cc:647] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:29.339581 11526 tablet_copy_client.cc:523] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:29.341141 11526 tablet_bootstrap.cc:492] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 01:59:29.379400 11526 tablet_bootstrap.cc:492] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:29.379837 11526 tablet_bootstrap.cc:492] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
I20240502 01:59:29.380100 11526 ts_tablet_manager.cc:1367] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.039s user 0.026s sys 0.002s
I20240502 01:59:29.380625 11526 raft_consensus.cc:348] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:29.380843 11526 raft_consensus.cc:729] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 01:59:29.381055 11526 consensus_queue.cc:260] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:29.382221 11526 ts_tablet_manager.cc:1398] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:29.382694 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-d4d429fe60e74108bc0c9ca8509365f2 received from {username='slave'} at 127.27.111.197:40005
I20240502 01:59:29.382768 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 954cd49326bf405abd81182d75dc783c-d4d429fe60e74108bc0c9ca8509365f2 on tablet d4d429fe60e74108bc0c9ca8509365f2 with peer 954cd49326bf405abd81182d75dc783c
I20240502 01:59:29.410753 8904 raft_consensus.cc:1203] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:29.425504 11397 tablet_bootstrap.cc:492] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:29.425925 11397 tablet_bootstrap.cc:492] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap complete.
I20240502 01:59:29.426138 11397 ts_tablet_manager.cc:1367] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent bootstrapping tablet: real 0.103s user 0.023s sys 0.004s
I20240502 01:59:29.426685 11397 raft_consensus.cc:348] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:29.426851 11397 raft_consensus.cc:729] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, State: Initialized, Role: FOLLOWER
I20240502 01:59:29.427062 11397 consensus_queue.cc:260] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:29.427469 11397 ts_tablet_manager.cc:1398] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:29.428102 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-b69d6e907b3b400f93721981732c006b received from {username='slave'} at 127.27.111.199:49969
I20240502 01:59:29.428184 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-b69d6e907b3b400f93721981732c006b on tablet b69d6e907b3b400f93721981732c006b with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:29.465898 6916 consensus_queue.cc:237] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:29.467437 7181 raft_consensus.cc:1261] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:29.468436 8509 raft_consensus.cc:1261] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:29.468521 11140 consensus_queue.cc:1035] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:29.470194 11140 consensus_queue.cc:1035] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:29.473981 11140 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:29.476926 8508 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:29.485692 6634 catalog_manager.cc:5484] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c reported cstate change: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:29.502722 7181 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:29.515468 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:29.516034 11696 tablet_replica.cc:330] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:29.516456 11696 raft_consensus.cc:2229] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:29.516762 11696 raft_consensus.cc:2258] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:29.517504 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.518112 11696 ts_tablet_manager.cc:1875] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:29.521047 11696 ts_tablet_manager.cc:1888] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:29.521368 11696 log.cc:1196] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/9ecbf1c8ee8c46b195d2374a5db81ac0
I20240502 01:59:29.522104 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 (table rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]) successfully deleted
I20240502 01:59:29.534034 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.534682 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.545063 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.546582 8760 raft_consensus.cc:1203] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:29.560237 8904 raft_consensus.cc:1203] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:59:29.566570 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:29.568617 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:29.572779 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:29.578442 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.590660 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.591413 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:29.599607 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.625698 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.626067 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:29.632041 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:29.633069 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:29.642719 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:29.665076 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.681080 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:29.690732 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.696857 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.720557 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.720970 8508 raft_consensus.cc:1203] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:59:29.721989 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.729949 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.732498 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.733839 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:29.736253 8508 raft_consensus.cc:1203] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:59:29.744688 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:29.773097 8760 raft_consensus.cc:1203] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:59:29.776161 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:29.793542 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:29.803674 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:29.805171 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:29.812497 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:29.818851 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.884667 7050 consensus_queue.cc:237] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:29.886322 8508 raft_consensus.cc:1261] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:29.886365 6785 raft_consensus.cc:1261] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:29.886947 10486 consensus_queue.cc:1035] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:29.887156 11731 consensus_queue.cc:1035] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:29.908342 6783 consensus_queue.cc:237] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:29.909314 7182 consensus_queue.cc:237] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:29.911043 7050 raft_consensus.cc:1261] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:29.911113 8904 raft_consensus.cc:1261] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:29.911449 6916 raft_consensus.cc:1261] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:29.912063 11732 consensus_queue.cc:1035] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:29.912083 11769 consensus_queue.cc:1035] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:29.912766 8760 raft_consensus.cc:1261] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:29.913652 11769 consensus_queue.cc:1035] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:29.914059 10486 raft_consensus.cc:2932] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:29.915293 11732 consensus_queue.cc:1035] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:29.917290 8508 raft_consensus.cc:2932] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:29.919126 11732 raft_consensus.cc:2932] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:29.920822 6637 catalog_manager.cc:5484] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:29.921972 8904 raft_consensus.cc:2932] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:29.923849 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet 9313079c48c94ab1b7e022a91befafcd with delete_type TABLET_DATA_TOMBSTONED (TS e7bcff2fef614e7a9981554c9e6db3c8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59160
I20240502 01:59:29.926697 6785 raft_consensus.cc:2932] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:29.926973 6635 catalog_manager.cc:5484] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:29.930244 11868 tablet_replica.cc:330] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8: stopping tablet replica
I20240502 01:59:29.931003 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.931991 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 4154be3f1f3c428580463c419d69892e with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:29.932790 11696 tablet_replica.cc:330] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:29.932945 11696 raft_consensus.cc:2229] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:29.933100 11696 raft_consensus.cc:2258] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:29.933460 7048 consensus_queue.cc:237] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:29.933758 11696 ts_tablet_manager.cc:1875] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:29.934908 8641 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:29.935693 10486 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:29.937348 11696 ts_tablet_manager.cc:1888] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:29.937640 7182 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:29.938041 11731 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:29.938444 11731 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:29.939097 11696 log.cc:1196] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/4154be3f1f3c428580463c419d69892e
I20240502 01:59:29.939728 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet 4154be3f1f3c428580463c419d69892e (table rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]) successfully deleted
I20240502 01:59:29.940253 8641 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:29.941355 6638 catalog_manager.cc:5484] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:29.944754 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a276e9780e448a9b3c6eab67925514 with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:29.944998 11696 tablet_replica.cc:330] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:29.945164 11696 raft_consensus.cc:2229] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:29.945312 11696 raft_consensus.cc:2258] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:29.945847 11696 ts_tablet_manager.cc:1875] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:29.948946 11696 ts_tablet_manager.cc:1888] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:29.949760 11696 log.cc:1196] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/47a276e9780e448a9b3c6eab67925514
I20240502 01:59:29.950806 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.951459 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet 47a276e9780e448a9b3c6eab67925514 (table rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]) successfully deleted
I20240502 01:59:29.957429 7182 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:29.959131 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:29.964783 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.982574 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:29.986467 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.994180 11868 raft_consensus.cc:2229] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:29.994477 11868 raft_consensus.cc:2258] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:29.995693 11868 ts_tablet_manager.cc:1875] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:29.999433 11768 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:30.000051 8760 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:30.003222 6638 catalog_manager.cc:5484] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 reported cstate change: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:30.005228 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:30.007999 7161 tablet_service.cc:1514] Processing DeleteTablet for tablet a4fb9af11abc49d496b5c3d8048c22c8 with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36602
I20240502 01:59:30.017544 6916 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:30.023311 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.025243 11868 ts_tablet_manager.cc:1888] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:30.025439 11868 log.cc:1196] T 9313079c48c94ab1b7e022a91befafcd P e7bcff2fef614e7a9981554c9e6db3c8: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/9313079c48c94ab1b7e022a91befafcd
I20240502 01:59:30.026173 6630 catalog_manager.cc:4830] TS e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): tablet 9313079c48c94ab1b7e022a91befafcd (table rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]) successfully deleted
I20240502 01:59:30.028584 7050 raft_consensus.cc:2932] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:30.034663 11917 tablet_replica.cc:330] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c: stopping tablet replica
I20240502 01:59:30.035050 11917 raft_consensus.cc:2229] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:30.035290 11917 raft_consensus.cc:2258] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:30.036451 11917 ts_tablet_manager.cc:1875] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:30.040302 11917 ts_tablet_manager.cc:1888] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:30.040462 11917 log.cc:1196] T a4fb9af11abc49d496b5c3d8048c22c8 P 86bc8001c3464b929cd5c873861baa0c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/a4fb9af11abc49d496b5c3d8048c22c8
I20240502 01:59:30.041214 6629 catalog_manager.cc:4830] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): tablet a4fb9af11abc49d496b5c3d8048c22c8 (table rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]) successfully deleted
I20240502 01:59:30.114526 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.119246 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.123979 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.128029 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.150261 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.154661 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:30.174194 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:30.189224 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:30.208478 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:30.214322 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.215749 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.226894 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:30.234814 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.240250 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:30.260178 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.269052 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.306788 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.326449 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.326449 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.328611 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.330600 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.339053 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:30.342638 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.348695 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:30.351040 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.373361 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:30.394560 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.411463 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.411768 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:30.420403 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.433358 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:30.451227 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:30.460940 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.466617 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.470492 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.474790 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.493129 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:30.500103 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.503647 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:30.533598 6785 consensus_queue.cc:237] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:30.534601 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:30.535187 8904 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:30.535681 12146 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:30.536295 6917 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:30.538870 11769 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:30.539407 11769 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:30.542909 8904 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:30.543664 6636 catalog_manager.cc:5484] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:30.548106 7050 consensus_queue.cc:237] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:30.549660 8508 raft_consensus.cc:1261] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:30.550598 6785 raft_consensus.cc:1261] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:30.551170 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet 0359dfbfd11746f1bf62a465f14634ec with delete_type TABLET_DATA_TOMBSTONED (TS c6101231591a458384f4ea6d1f0e0ae5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36030
I20240502 01:59:30.551368 8760 raft_consensus.cc:1261] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:30.551744 12164 tablet_replica.cc:330] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5: stopping tablet replica
I20240502 01:59:30.552011 11882 consensus_queue.cc:1035] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:30.552238 12164 raft_consensus.cc:2229] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:30.552480 12164 raft_consensus.cc:2258] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:30.552968 11731 consensus_queue.cc:1035] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:30.553227 11731 consensus_queue.cc:1035] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:30.554107 12164 ts_tablet_manager.cc:1875] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:30.557832 12164 ts_tablet_manager.cc:1888] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:30.558491 12164 log.cc:1196] T 0359dfbfd11746f1bf62a465f14634ec P c6101231591a458384f4ea6d1f0e0ae5: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/0359dfbfd11746f1bf62a465f14634ec
I20240502 01:59:30.559212 6630 catalog_manager.cc:4830] TS c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125): tablet 0359dfbfd11746f1bf62a465f14634ec (table rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]) successfully deleted
I20240502 01:59:30.560410 12163 raft_consensus.cc:2932] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:30.563289 8508 raft_consensus.cc:2932] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:30.566061 6641 catalog_manager.cc:5484] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 5 to 6, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:30.568501 8760 raft_consensus.cc:2932] T b69d6e907b3b400f93721981732c006b P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:30.573262 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet b69d6e907b3b400f93721981732c006b with delete_type TABLET_DATA_TOMBSTONED (TS e7bcff2fef614e7a9981554c9e6db3c8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59160
I20240502 01:59:30.575145 12170 tablet_replica.cc:330] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8: stopping tablet replica
I20240502 01:59:30.575340 12170 raft_consensus.cc:2229] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:30.575536 12170 raft_consensus.cc:2258] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:30.576210 12170 ts_tablet_manager.cc:1875] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:30.578745 12170 ts_tablet_manager.cc:1888] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:59:30.578886 12170 log.cc:1196] T b69d6e907b3b400f93721981732c006b P e7bcff2fef614e7a9981554c9e6db3c8: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/b69d6e907b3b400f93721981732c006b
I20240502 01:59:30.579000 6917 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:30.579429 6630 catalog_manager.cc:4830] TS e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): tablet b69d6e907b3b400f93721981732c006b (table rebalance_test_table_0 [id=6397b955acf741aeb8f869ab987d74b0]) successfully deleted
I20240502 01:59:30.582640 6785 raft_consensus.cc:2932] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:30.596772 6785 consensus_queue.cc:237] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:30.598412 8904 raft_consensus.cc:1261] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:30.598661 6917 raft_consensus.cc:1261] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:30.598886 11769 consensus_queue.cc:1035] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:30.599058 12146 consensus_queue.cc:1035] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:30.610881 12146 raft_consensus.cc:2932] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:30.611307 8904 raft_consensus.cc:2932] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:30.613559 6916 raft_consensus.cc:2932] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:30.614101 6639 catalog_manager.cc:5484] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d reported cstate change: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:30.619261 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f6c07e3923d47f29ef32561d7f438e1 with delete_type TABLET_DATA_TOMBSTONED (TS c6101231591a458384f4ea6d1f0e0ae5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36030
I20240502 01:59:30.619513 12164 tablet_replica.cc:330] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5: stopping tablet replica
I20240502 01:59:30.619639 12164 raft_consensus.cc:2229] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:30.619760 12164 raft_consensus.cc:2258] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:30.620334 12164 ts_tablet_manager.cc:1875] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:30.626165 12164 ts_tablet_manager.cc:1888] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:30.627979 6916 consensus_queue.cc:237] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:30.629761 8508 raft_consensus.cc:1261] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:30.630467 12164 log.cc:1196] T 9f6c07e3923d47f29ef32561d7f438e1 P c6101231591a458384f4ea6d1f0e0ae5: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/9f6c07e3923d47f29ef32561d7f438e1
I20240502 01:59:30.630451 11920 consensus_queue.cc:1035] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:30.631804 6630 catalog_manager.cc:4830] TS c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125): tablet 9f6c07e3923d47f29ef32561d7f438e1 (table rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]) successfully deleted
I20240502 01:59:30.639668 11920 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:30.642370 6916 consensus_queue.cc:237] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:30.642694 6636 catalog_manager.cc:5484] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:30.643996 8508 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:30.646093 7161 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36602
I20240502 01:59:30.646607 8642 raft_consensus.cc:1261] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:30.647037 7050 raft_consensus.cc:1261] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:30.647207 6916 consensus_queue.cc:237] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:30.647536 8508 raft_consensus.cc:1261] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:30.648527 12248 consensus_queue.cc:1035] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:30.648813 8508 raft_consensus.cc:1261] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:59:30.649247 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ecbf1c8ee8c46b195d2374a5db81ac0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:30.650817 11920 consensus_queue.cc:1035] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:30.653080 11920 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:30.653355 8508 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:30.655305 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:59:30.652762 12254 consensus_queue.cc:1035] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:30.655782 11140 consensus_queue.cc:1035] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:30.656205 6636 catalog_manager.cc:5484] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 6 to 7, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:30.659377 11140 raft_consensus.cc:2932] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:30.659837 7050 raft_consensus.cc:2932] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:30.661285 12249 tablet_replica.cc:330] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c: stopping tablet replica
I20240502 01:59:30.661494 12249 raft_consensus.cc:2229] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:30.662000 8508 raft_consensus.cc:2932] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:30.662102 6636 catalog_manager.cc:5484] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:30.663545 8642 raft_consensus.cc:2932] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:30.661692 12249 raft_consensus.cc:2258] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:30.667274 7161 tablet_service.cc:1514] Processing DeleteTablet for tablet 17235e019e644b258266636fb6be6648 with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36602
I20240502 01:59:30.667695 12249 ts_tablet_manager.cc:1875] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:30.670760 12249 ts_tablet_manager.cc:1888] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:59:30.670902 12249 log.cc:1196] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 86bc8001c3464b929cd5c873861baa0c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/9ecbf1c8ee8c46b195d2374a5db81ac0
I20240502 01:59:30.671439 12249 tablet_replica.cc:330] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c: stopping tablet replica
I20240502 01:59:30.671500 6629 catalog_manager.cc:4830] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 (table rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]) successfully deleted
I20240502 01:59:30.671661 12249 raft_consensus.cc:2229] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:30.671838 12249 raft_consensus.cc:2258] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:30.672385 12249 ts_tablet_manager.cc:1875] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:30.674491 12249 ts_tablet_manager.cc:1888] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:59:30.677080 12249 log.cc:1196] T 17235e019e644b258266636fb6be6648 P 86bc8001c3464b929cd5c873861baa0c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/17235e019e644b258266636fb6be6648
I20240502 01:59:30.677947 6629 catalog_manager.cc:4830] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): tablet 17235e019e644b258266636fb6be6648 (table rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]) successfully deleted
I20240502 01:59:30.744935 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.752517 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.752661 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:30.760253 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:30.767554 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.772744 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.807360 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.808895 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.830905 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:30.833042 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.834234 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:30.838624 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.841585 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.842239 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:30.851526 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:30.857685 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.875478 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.875929 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.882661 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.886570 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.888875 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.906090 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:30.926616 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:30.968986 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:30.990607 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.007035 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:31.013994 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.019950 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:31.020856 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:31.022178 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.066262 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:31.072602 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.083163 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.083210 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.085922 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:31.090952 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:31.114377 7181 consensus_queue.cc:237] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:31.115753 6916 raft_consensus.cc:1261] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:31.116185 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.116303 11412 consensus_queue.cc:1035] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:31.117522 8903 raft_consensus.cc:1261] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:31.118507 11412 consensus_queue.cc:1035] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:31.120427 11412 raft_consensus.cc:2932] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:31.123852 8903 raft_consensus.cc:2932] T 347b161e9ec84640a0668836ce9e29eb P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:31.127570 6641 catalog_manager.cc:5484] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:31.130092 6916 raft_consensus.cc:2932] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:31.131943 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 347b161e9ec84640a0668836ce9e29eb with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:31.138469 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:31.139122 12479 tablet_replica.cc:330] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:31.139295 12479 raft_consensus.cc:2229] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:31.139490 12479 raft_consensus.cc:2258] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:31.140079 12479 ts_tablet_manager.cc:1875] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:31.140641 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.141708 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:31.143242 12479 ts_tablet_manager.cc:1888] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:31.143324 12479 log.cc:1196] T 347b161e9ec84640a0668836ce9e29eb P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/347b161e9ec84640a0668836ce9e29eb
I20240502 01:59:31.143895 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet 347b161e9ec84640a0668836ce9e29eb (table rebalance_test_table_9 [id=19f17ecb48ac4307bcd5af6811f21abe]) successfully deleted
I20240502 01:59:31.172187 12501 ts_tablet_manager.cc:902] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:31.172919 12501 tablet_copy_client.cc:308] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 01:59:31.173610 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 from peer 070558c278e0457b9cc070ac0d662b5d ({username='slave'} at 127.27.111.200:54661)
I20240502 01:59:31.173743 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 from peer 070558c278e0457b9cc070ac0d662b5d at {username='slave'} at 127.27.111.200:54661: session id = 070558c278e0457b9cc070ac0d662b5d-9ecbf1c8ee8c46b195d2374a5db81ac0
I20240502 01:59:31.174896 6937 tablet_copy_source_session.cc:205] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:31.176050 12501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:31.178494 12501 tablet_copy_client.cc:783] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:31.178655 12501 tablet_copy_client.cc:647] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:31.179870 12501 tablet_copy_client.cc:523] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:31.181257 12501 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap starting.
I20240502 01:59:31.183430 6916 consensus_queue.cc:237] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:31.185084 8760 raft_consensus.cc:1261] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:31.185528 7181 raft_consensus.cc:1261] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:31.186213 12255 consensus_queue.cc:1035] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:31.186453 12509 consensus_queue.cc:1035] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:31.188424 12255 raft_consensus.cc:2932] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:31.189988 7181 raft_consensus.cc:2932] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:31.191278 6637 catalog_manager.cc:5484] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:31.198673 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet ed7b67af7b1c4255a7c2377570b58e70 with delete_type TABLET_DATA_TOMBSTONED (TS c6101231591a458384f4ea6d1f0e0ae5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36030
I20240502 01:59:31.199123 12521 tablet_replica.cc:330] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5: stopping tablet replica
I20240502 01:59:31.199328 12521 raft_consensus.cc:2229] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:31.199564 12521 raft_consensus.cc:2258] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:31.204295 7050 consensus_queue.cc:237] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:31.207036 12521 ts_tablet_manager.cc:1875] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:31.208061 6785 raft_consensus.cc:1261] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:31.208621 8760 raft_consensus.cc:2932] T ed7b67af7b1c4255a7c2377570b58e70 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:31.208930 8508 raft_consensus.cc:1261] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:31.209745 12521 ts_tablet_manager.cc:1888] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:31.209825 12521 log.cc:1196] T ed7b67af7b1c4255a7c2377570b58e70 P c6101231591a458384f4ea6d1f0e0ae5: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/ed7b67af7b1c4255a7c2377570b58e70
I20240502 01:59:31.209945 11882 consensus_queue.cc:1035] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:31.210260 8903 raft_consensus.cc:1261] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:31.210647 6630 catalog_manager.cc:4830] TS c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125): tablet ed7b67af7b1c4255a7c2377570b58e70 (table rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]) successfully deleted
I20240502 01:59:31.211122 12501 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:31.211405 12501 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap complete.
I20240502 01:59:31.211562 12501 ts_tablet_manager.cc:1367] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.004s
I20240502 01:59:31.211592 11731 consensus_queue.cc:1035] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:31.211882 11731 consensus_queue.cc:1035] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:31.211992 12501 raft_consensus.cc:348] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:31.212090 12501 raft_consensus.cc:729] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Initialized, Role: FOLLOWER
I20240502 01:59:31.212242 12501 consensus_queue.cc:260] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:31.212603 12501 ts_tablet_manager.cc:1398] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:31.213124 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 070558c278e0457b9cc070ac0d662b5d-9ecbf1c8ee8c46b195d2374a5db81ac0 received from {username='slave'} at 127.27.111.200:54661
I20240502 01:59:31.213579 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 070558c278e0457b9cc070ac0d662b5d-9ecbf1c8ee8c46b195d2374a5db81ac0 on tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 with peer 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:31.220645 11882 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:31.221066 8508 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:31.226001 6783 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:31.226590 6637 catalog_manager.cc:5484] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:31.227458 8903 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:31.233173 7161 tablet_service.cc:1514] Processing DeleteTablet for tablet 7345f51bd6f84b3880838c2460bafb75 with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36602
I20240502 01:59:31.234838 12541 tablet_replica.cc:330] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c: stopping tablet replica
I20240502 01:59:31.237085 12541 raft_consensus.cc:2229] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:31.238359 12541 raft_consensus.cc:2258] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:31.240607 12541 ts_tablet_manager.cc:1875] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:31.273047 12541 ts_tablet_manager.cc:1888] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:59:31.273252 12541 log.cc:1196] T 7345f51bd6f84b3880838c2460bafb75 P 86bc8001c3464b929cd5c873861baa0c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/7345f51bd6f84b3880838c2460bafb75
I20240502 01:59:31.276026 6629 catalog_manager.cc:4830] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): tablet 7345f51bd6f84b3880838c2460bafb75 (table rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]) successfully deleted
I20240502 01:59:31.280778 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.293774 7050 consensus_queue.cc:237] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:31.295703 6783 raft_consensus.cc:1261] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:31.295804 8760 raft_consensus.cc:1261] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:31.296240 11731 consensus_queue.cc:1035] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:31.296552 11731 consensus_queue.cc:1035] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:31.306710 7050 consensus_queue.cc:237] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:31.308382 8509 raft_consensus.cc:1261] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:31.308487 6916 raft_consensus.cc:1261] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:31.308866 12537 consensus_queue.cc:1035] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:31.309167 12528 consensus_queue.cc:1035] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:31.312206 12528 raft_consensus.cc:2932] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:31.315650 6637 catalog_manager.cc:5484] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:31.316200 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:31.317459 11731 raft_consensus.cc:2932] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:31.319319 6783 raft_consensus.cc:2932] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:31.320794 8760 raft_consensus.cc:2932] T e0e05f96dbc542eb999c16dcc0620666 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:31.323859 6640 catalog_manager.cc:5484] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:31.324703 6916 raft_consensus.cc:2932] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:31.326696 7161 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d429fe60e74108bc0c9ca8509365f2 with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36602
I20240502 01:59:31.326920 12541 tablet_replica.cc:330] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: stopping tablet replica
I20240502 01:59:31.327059 12541 raft_consensus.cc:2229] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:31.327217 12541 raft_consensus.cc:2258] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:31.327961 12541 ts_tablet_manager.cc:1875] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:31.331473 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet e0e05f96dbc542eb999c16dcc0620666 with delete_type TABLET_DATA_TOMBSTONED (TS e7bcff2fef614e7a9981554c9e6db3c8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59160
I20240502 01:59:31.331699 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:31.331967 12636 tablet_replica.cc:330] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8: stopping tablet replica
I20240502 01:59:31.332175 12636 raft_consensus.cc:2229] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:31.333050 12636 raft_consensus.cc:2258] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:31.333848 12636 ts_tablet_manager.cc:1875] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:31.334220 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.337855 12636 ts_tablet_manager.cc:1888] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:31.338014 12636 log.cc:1196] T e0e05f96dbc542eb999c16dcc0620666 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/e0e05f96dbc542eb999c16dcc0620666
I20240502 01:59:31.338635 6630 catalog_manager.cc:4830] TS e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): tablet e0e05f96dbc542eb999c16dcc0620666 (table rebalance_test_table_4 [id=f2b695d4b3e54fdfad81f019e9ab7b4a]) successfully deleted
I20240502 01:59:31.343247 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.344098 12541 ts_tablet_manager.cc:1888] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:31.344185 12541 log.cc:1196] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/d4d429fe60e74108bc0c9ca8509365f2
I20240502 01:59:31.344779 6629 catalog_manager.cc:4830] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): tablet d4d429fe60e74108bc0c9ca8509365f2 (table rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]) successfully deleted
I20240502 01:59:31.345839 8509 raft_consensus.cc:2932] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:31.357748 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:31.377178 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:31.381367 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.442157 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.453691 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:31.455494 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.458254 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.462603 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.495705 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:31.496758 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:31.499173 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:31.562227 6783 consensus_queue.cc:237] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:31.563800 7050 raft_consensus.cc:1261] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:31.564285 11769 consensus_queue.cc:1035] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:31.564437 8509 raft_consensus.cc:1261] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:31.565472 12146 consensus_queue.cc:1035] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:31.569653 12146 raft_consensus.cc:2932] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:31.570339 8509 raft_consensus.cc:2932] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:31.573199 6636 catalog_manager.cc:5484] T 4e44267bd266407f9b72c66e3d3579bb P 954cd49326bf405abd81182d75dc783c reported cstate change: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:31.576702 7161 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e44267bd266407f9b72c66e3d3579bb with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36602
I20240502 01:59:31.576959 12541 tablet_replica.cc:330] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c: stopping tablet replica
I20240502 01:59:31.577129 12541 raft_consensus.cc:2229] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:31.577297 12541 raft_consensus.cc:2258] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:31.577919 12541 ts_tablet_manager.cc:1875] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:31.584906 12541 ts_tablet_manager.cc:1888] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:31.587200 12541 log.cc:1196] T 4e44267bd266407f9b72c66e3d3579bb P 86bc8001c3464b929cd5c873861baa0c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/4e44267bd266407f9b72c66e3d3579bb
I20240502 01:59:31.588207 6629 catalog_manager.cc:4830] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): tablet 4e44267bd266407f9b72c66e3d3579bb (table rebalance_test_table_9 [id=19f17ecb48ac4307bcd5af6811f21abe]) successfully deleted
I20240502 01:59:31.589627 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:31.597246 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.606691 7050 raft_consensus.cc:2932] T 4e44267bd266407f9b72c66e3d3579bb P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:31.608217 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.614077 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.617208 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:31.617547 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.620868 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:31.640384 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.644426 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:31.649039 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.653928 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:31.657142 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:31.666576 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:31.677984 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:31.688903 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.698606 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.758915 8903 raft_consensus.cc:1203] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:59:31.769008 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.777387 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.779240 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:31.792856 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.807879 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:31.809194 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:31.820255 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:31.836441 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.876526 6783 consensus_queue.cc:237] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:31.878064 8760 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:31.878640 11769 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:31.878793 7050 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:31.879351 11769 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:31.882875 11769 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:31.886150 8760 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:31.887068 7050 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:31.889159 6641 catalog_manager.cc:5484] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:31.893384 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet a0548274c47148d9b611fd3c978a0c34 with delete_type TABLET_DATA_TOMBSTONED (TS e7bcff2fef614e7a9981554c9e6db3c8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59160
I20240502 01:59:31.895328 7181 consensus_queue.cc:237] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:31.895743 12928 tablet_replica.cc:330] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8: stopping tablet replica
I20240502 01:59:31.895893 12928 raft_consensus.cc:2229] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:31.896039 12928 raft_consensus.cc:2258] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:31.896557 12928 ts_tablet_manager.cc:1875] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:31.897275 8903 raft_consensus.cc:1261] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:31.897275 8509 raft_consensus.cc:1261] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:31.897940 11412 consensus_queue.cc:1035] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:31.898330 11412 consensus_queue.cc:1035] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:31.899124 12928 ts_tablet_manager.cc:1888] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:31.899209 12928 log.cc:1196] T a0548274c47148d9b611fd3c978a0c34 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/a0548274c47148d9b611fd3c978a0c34
I20240502 01:59:31.899753 6630 catalog_manager.cc:4830] TS e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): tablet a0548274c47148d9b611fd3c978a0c34 (table rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]) successfully deleted
I20240502 01:59:31.900310 6917 raft_consensus.cc:1261] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:31.903067 12936 consensus_queue.cc:1035] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:31.906049 12937 raft_consensus.cc:2932] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:31.906250 6917 raft_consensus.cc:2932] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:31.906806 8509 raft_consensus.cc:2932] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:31.910333 6637 catalog_manager.cc:5484] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c reported cstate change: config changed from index 5 to 6, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:31.910588 8903 raft_consensus.cc:2932] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:31.917840 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c54b912f802461c8cac092ac3222c87 with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:31.923342 12951 tablet_replica.cc:330] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:31.924074 12951 raft_consensus.cc:2229] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:31.925666 12951 raft_consensus.cc:2258] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:31.926296 12951 ts_tablet_manager.cc:1875] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:31.928836 12951 ts_tablet_manager.cc:1888] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:59:31.928918 12951 log.cc:1196] T 1c54b912f802461c8cac092ac3222c87 P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/1c54b912f802461c8cac092ac3222c87
I20240502 01:59:31.929548 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet 1c54b912f802461c8cac092ac3222c87 (table rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]) successfully deleted
I20240502 01:59:31.941017 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.950452 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.963995 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.966540 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:31.966915 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.970402 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:32.001798 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:32.003870 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.013967 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.014315 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:32.021397 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:32.024088 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.043206 6783 consensus_queue.cc:237] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:32.044845 7181 raft_consensus.cc:1261] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:32.045264 12916 consensus_queue.cc:1035] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:32.046663 8642 raft_consensus.cc:1261] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:32.047518 12916 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.049502 7181 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.052294 6641 catalog_manager.cc:5484] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.055454 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet c8c237c50773494abf8813494be0f184 with delete_type TABLET_DATA_TOMBSTONED (TS c6101231591a458384f4ea6d1f0e0ae5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36030
I20240502 01:59:32.056005 11769 consensus_queue.cc:1035] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:32.057379 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:32.058276 8642 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.073643 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:32.074496 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:32.104310 13031 tablet_replica.cc:330] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5: stopping tablet replica
I20240502 01:59:32.104524 13031 raft_consensus.cc:2229] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:32.104687 13031 raft_consensus.cc:2258] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:32.105224 13031 ts_tablet_manager.cc:1875] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:32.107796 13031 ts_tablet_manager.cc:1888] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:32.107939 13031 log.cc:1196] T c8c237c50773494abf8813494be0f184 P c6101231591a458384f4ea6d1f0e0ae5: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/c8c237c50773494abf8813494be0f184
I20240502 01:59:32.108536 6630 catalog_manager.cc:4830] TS c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125): tablet c8c237c50773494abf8813494be0f184 (table rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]) successfully deleted
I20240502 01:59:32.109023 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.153970 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.165076 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.166551 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.177626 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.185087 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:32.192337 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:32.201185 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:32.212430 6917 consensus_queue.cc:237] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:32.213979 7181 raft_consensus.cc:1261] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:32.214066 8903 raft_consensus.cc:1261] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:32.214637 12255 consensus_queue.cc:1035] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:32.214965 12255 consensus_queue.cc:1035] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:32.215041 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:32.223310 12255 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:32.225536 7181 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:32.228328 6641 catalog_manager.cc:5484] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:32.233973 6916 consensus_queue.cc:237] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:32.235602 7050 raft_consensus.cc:1261] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:32.236119 13123 consensus_queue.cc:1035] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:32.236555 8760 raft_consensus.cc:1261] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:32.236925 13123 consensus_queue.cc:1035] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:32.239353 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 39d9e48f67f34ebe850fc388ecd94ca3 with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:32.239574 12951 tablet_replica.cc:330] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:32.239708 12951 raft_consensus.cc:2229] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:32.239881 12951 raft_consensus.cc:2258] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:32.240483 12951 ts_tablet_manager.cc:1875] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:32.243499 12951 ts_tablet_manager.cc:1888] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:32.243634 12951 log.cc:1196] T 39d9e48f67f34ebe850fc388ecd94ca3 P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/39d9e48f67f34ebe850fc388ecd94ca3
I20240502 01:59:32.244446 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet 39d9e48f67f34ebe850fc388ecd94ca3 (table rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]) successfully deleted
I20240502 01:59:32.263723 13123 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:32.264096 7048 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:32.267120 6641 catalog_manager.cc:5484] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:32.267417 8904 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:32.270035 8760 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:32.277236 7161 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc5daef41bc44c4855f2bcb44e1c6d5 with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36602
I20240502 01:59:32.281683 13145 tablet_replica.cc:330] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c: stopping tablet replica
I20240502 01:59:32.281941 13145 raft_consensus.cc:2229] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:32.282193 13145 raft_consensus.cc:2258] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:32.282940 13145 ts_tablet_manager.cc:1875] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:32.286073 13145 ts_tablet_manager.cc:1888] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:32.286765 13145 log.cc:1196] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 86bc8001c3464b929cd5c873861baa0c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/dcc5daef41bc44c4855f2bcb44e1c6d5
I20240502 01:59:32.287869 6629 catalog_manager.cc:4830] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): tablet dcc5daef41bc44c4855f2bcb44e1c6d5 (table rebalance_test_table_0 [id=6397b955acf741aeb8f869ab987d74b0]) successfully deleted
I20240502 01:59:32.346053 7048 consensus_queue.cc:237] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:32.348853 7181 raft_consensus.cc:1261] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:32.349522 8642 raft_consensus.cc:1261] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:32.350113 12537 consensus_queue.cc:1035] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:32.351349 12537 consensus_queue.cc:1035] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:32.376564 12528 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.376724 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.377431 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.379074 7181 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.381983 6636 catalog_manager.cc:5484] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.385656 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.388514 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet 14e3c78c65334b52aca79d67c4bf45a2 with delete_type TABLET_DATA_TOMBSTONED (TS e7bcff2fef614e7a9981554c9e6db3c8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59160
I20240502 01:59:32.388764 12928 tablet_replica.cc:330] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: stopping tablet replica
I20240502 01:59:32.388913 12928 raft_consensus.cc:2229] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:32.389083 12928 raft_consensus.cc:2258] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:32.389746 12928 ts_tablet_manager.cc:1875] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:32.392123 12928 ts_tablet_manager.cc:1888] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:32.392187 12928 log.cc:1196] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/14e3c78c65334b52aca79d67c4bf45a2
I20240502 01:59:32.392704 6630 catalog_manager.cc:4830] TS e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): tablet 14e3c78c65334b52aca79d67c4bf45a2 (table rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]) successfully deleted
I20240502 01:59:32.395924 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:32.396961 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.400985 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:32.404095 8642 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.404649 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.408869 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:32.413538 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.434640 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.438548 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:32.449717 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:32.450807 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.451624 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.487361 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.491741 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:32.501626 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:32.504300 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:32.505688 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.516305 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.528165 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:32.532740 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:32.534561 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:32.566242 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:32.634254 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.637168 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:32.638635 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.644709 7048 consensus_queue.cc:237] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:32.647022 8509 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:32.647413 6783 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:32.647881 12537 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:32.648214 8642 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:32.648238 12528 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:32.648304 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.648818 12528 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:32.652624 13370 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.654750 8760 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:32.655292 12528 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:32.655927 8642 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.658259 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.660277 6640 catalog_manager.cc:5484] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 6 to 7, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.660935 8757 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.663820 7161 tablet_service.cc:1514] Processing DeleteTablet for tablet 48eeb721a60c4be888041bd31cfeab7a with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36602
I20240502 01:59:32.664026 13145 tablet_replica.cc:330] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c: stopping tablet replica
I20240502 01:59:32.664144 13145 raft_consensus.cc:2229] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:32.664289 13145 raft_consensus.cc:2258] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:32.664875 13145 ts_tablet_manager.cc:1875] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:32.667732 13145 ts_tablet_manager.cc:1888] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:59:32.667888 13145 log.cc:1196] T 48eeb721a60c4be888041bd31cfeab7a P 86bc8001c3464b929cd5c873861baa0c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/48eeb721a60c4be888041bd31cfeab7a
I20240502 01:59:32.671625 6629 catalog_manager.cc:4830] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): tablet 48eeb721a60c4be888041bd31cfeab7a (table rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]) successfully deleted
I20240502 01:59:32.672124 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:32.672139 8509 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.685179 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:32.692513 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:32.699067 6783 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.716521 7048 consensus_queue.cc:237] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:32.718219 8642 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:32.718668 13370 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:32.720511 6783 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:32.721724 13370 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:32.724392 13370 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.727185 6783 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.729458 6640 catalog_manager.cc:5484] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.732508 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet 034cf1c3d47a4f7785c16d16815c229d with delete_type TABLET_DATA_TOMBSTONED (TS e7bcff2fef614e7a9981554c9e6db3c8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59160
I20240502 01:59:32.732738 12928 tablet_replica.cc:330] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8: stopping tablet replica
I20240502 01:59:32.732872 12928 raft_consensus.cc:2229] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:32.733013 12928 raft_consensus.cc:2258] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:32.733564 12928 ts_tablet_manager.cc:1875] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:32.735818 12928 ts_tablet_manager.cc:1888] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:32.735890 12928 log.cc:1196] T 034cf1c3d47a4f7785c16d16815c229d P e7bcff2fef614e7a9981554c9e6db3c8: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/034cf1c3d47a4f7785c16d16815c229d
I20240502 01:59:32.736349 6630 catalog_manager.cc:4830] TS e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): tablet 034cf1c3d47a4f7785c16d16815c229d (table rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]) successfully deleted
I20240502 01:59:32.748487 8642 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.755550 6917 consensus_queue.cc:237] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:32.757563 8642 raft_consensus.cc:1261] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:32.757843 7181 raft_consensus.cc:1261] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:32.758265 13116 consensus_queue.cc:1035] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:32.758437 12255 consensus_queue.cc:1035] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:32.765281 13116 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.768859 8641 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.772773 6639 catalog_manager.cc:5484] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.775955 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet 818dbc9a8e054eeaa122e7919a311806 with delete_type TABLET_DATA_TOMBSTONED (TS c6101231591a458384f4ea6d1f0e0ae5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36030
I20240502 01:59:32.784181 7181 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:32.794442 13432 tablet_replica.cc:330] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: stopping tablet replica
I20240502 01:59:32.795917 13432 raft_consensus.cc:2229] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:32.796209 13432 raft_consensus.cc:2258] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:32.796900 6783 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c8c237c50773494abf8813494be0f184"
dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48134
I20240502 01:59:32.797078 6783 raft_consensus.cc:595] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Received request to transfer leadership
I20240502 01:59:32.796959 13432 ts_tablet_manager.cc:1875] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:32.821581 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.824776 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:32.824918 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.826725 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.828372 13432 ts_tablet_manager.cc:1888] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:32.831290 13432 log.cc:1196] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/818dbc9a8e054eeaa122e7919a311806
I20240502 01:59:32.833662 6630 catalog_manager.cc:4830] TS c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125): tablet 818dbc9a8e054eeaa122e7919a311806 (table rebalance_test_table_4 [id=f2b695d4b3e54fdfad81f019e9ab7b4a]) successfully deleted
I20240502 01:59:32.860000 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:32.872001 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:32.874521 8621 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.880059 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:32.989140 12146 raft_consensus.cc:979] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e: : Instructing follower 4f53d80e39be410d92d4766692aff1c6 to start an election
I20240502 01:59:32.989259 12146 raft_consensus.cc:1067] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Signalling peer 4f53d80e39be410d92d4766692aff1c6 to start an election
I20240502 01:59:32.989849 8642 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8c237c50773494abf8813494be0f184"
dest_uuid: "4f53d80e39be410d92d4766692aff1c6"
from {username='slave'} at 127.27.111.193:36281
I20240502 01:59:32.990077 8642 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:32.990185 8642 raft_consensus.cc:3037] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:32.991397 8642 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:32.992596 8642 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [CANDIDATE]: Term 2 election: Requested vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:32.994310 6783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c237c50773494abf8813494be0f184" candidate_uuid: "4f53d80e39be410d92d4766692aff1c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:32.994465 6783 raft_consensus.cc:3032] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:32.994513 6783 raft_consensus.cc:729] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:32.994683 6783 consensus_queue.cc:260] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:32.994854 6783 raft_consensus.cc:3037] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:32.996011 6783 raft_consensus.cc:2454] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f53d80e39be410d92d4766692aff1c6 in term 2.
I20240502 01:59:32.996767 8576 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f53d80e39be410d92d4766692aff1c6, 85e78945910a4e1abfa101d2edc3644e; no voters:
I20240502 01:59:32.997040 13033 raft_consensus.cc:2781] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:32.997246 13033 raft_consensus.cc:686] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [term 2 LEADER]: Becoming Leader. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Running, Role: LEADER
I20240502 01:59:32.997561 13033 consensus_queue.cc:237] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:33.000411 6642 catalog_manager.cc:5484] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: term changed from 1 to 2, leader changed from 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) to 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198). New cstate: current_term: 2 leader_uuid: "4f53d80e39be410d92d4766692aff1c6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:33.036113 7181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c237c50773494abf8813494be0f184" candidate_uuid: "4f53d80e39be410d92d4766692aff1c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:33.036309 7181 raft_consensus.cc:3037] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:33.037597 7181 raft_consensus.cc:2454] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f53d80e39be410d92d4766692aff1c6 in term 2.
I20240502 01:59:33.047289 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.049101 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.064045 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:33.065747 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.075459 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:33.076393 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:33.080812 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:33.083773 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:33.090378 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.103971 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.123358 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.129511 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.139966 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:33.140687 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.140712 7182 consensus_queue.cc:237] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:33.142074 6916 raft_consensus.cc:1261] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:33.142163 8904 raft_consensus.cc:1261] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:33.142695 13584 consensus_queue.cc:1035] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:33.142928 12934 consensus_queue.cc:1035] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:33.154920 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.162822 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.197818 12934 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:33.200349 8904 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:33.205572 6642 catalog_manager.cc:5484] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:33.207449 6916 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:33.210801 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fa358140eb742b587a4d6a203738b4f with delete_type TABLET_DATA_TOMBSTONED (TS c6101231591a458384f4ea6d1f0e0ae5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36030
I20240502 01:59:33.211174 13432 tablet_replica.cc:330] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5: stopping tablet replica
I20240502 01:59:33.211314 13432 raft_consensus.cc:2229] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:33.211500 13432 raft_consensus.cc:2258] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:33.212131 13432 ts_tablet_manager.cc:1875] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:33.214535 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.215580 13432 ts_tablet_manager.cc:1888] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:33.215718 13432 log.cc:1196] T 7fa358140eb742b587a4d6a203738b4f P c6101231591a458384f4ea6d1f0e0ae5: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/7fa358140eb742b587a4d6a203738b4f
I20240502 01:59:33.216305 6630 catalog_manager.cc:4830] TS c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125): tablet 7fa358140eb742b587a4d6a203738b4f (table rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]) successfully deleted
I20240502 01:59:33.222703 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.238422 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:33.238549 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.238850 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:33.252341 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.256145 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.269812 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:33.282286 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.291095 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:33.295984 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.296854 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.327812 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:33.338093 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.342139 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:33.349442 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.386641 7048 consensus_queue.cc:237] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:33.389202 8508 raft_consensus.cc:1261] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:33.389492 8904 raft_consensus.cc:1261] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:33.389734 12528 consensus_queue.cc:1035] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:33.390425 12528 consensus_queue.cc:1035] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:33.392853 13370 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:33.393241 8904 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:33.395931 6636 catalog_manager.cc:5484] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 6 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:33.399945 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 7345f51bd6f84b3880838c2460bafb75 with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:33.401537 8508 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:33.402139 13714 tablet_replica.cc:330] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:33.402266 13714 raft_consensus.cc:2229] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:33.402494 13714 raft_consensus.cc:2258] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:33.403179 13714 ts_tablet_manager.cc:1875] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:33.407336 13714 ts_tablet_manager.cc:1888] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:59:33.407480 13714 log.cc:1196] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/7345f51bd6f84b3880838c2460bafb75
I20240502 01:59:33.408183 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet 7345f51bd6f84b3880838c2460bafb75 (table rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]) successfully deleted
I20240502 01:59:33.423331 6783 consensus_queue.cc:237] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:33.425037 7182 raft_consensus.cc:1261] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:33.425037 8641 raft_consensus.cc:1261] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:33.425475 12146 consensus_queue.cc:1035] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:33.425662 13570 consensus_queue.cc:1035] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:33.443876 7181 raft_consensus.cc:1261] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:59:33.444407 13572 consensus_queue.cc:1035] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:33.446787 13570 raft_consensus.cc:2932] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:33.447636 7182 consensus_queue.cc:237] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:33.449378 8757 raft_consensus.cc:1261] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:33.449399 6783 raft_consensus.cc:1261] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:33.449795 13584 consensus_queue.cc:1035] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:33.450546 7182 raft_consensus.cc:2932] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:33.452575 13584 consensus_queue.cc:1035] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:33.452935 8641 raft_consensus.cc:2932] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:33.453027 6637 catalog_manager.cc:5484] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:33.456801 6785 raft_consensus.cc:1261] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:59:33.457985 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet a66b932d884e4df8a78841103d88ded0 with delete_type TABLET_DATA_TOMBSTONED (TS e7bcff2fef614e7a9981554c9e6db3c8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59160
I20240502 01:59:33.461397 13584 raft_consensus.cc:2932] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:33.462304 8757 raft_consensus.cc:2932] T 8529a93abd6e4d62bbc0bddcf2fda78a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:33.466718 6783 raft_consensus.cc:2932] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:33.470057 6642 catalog_manager.cc:5484] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 4 to 5, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:33.474773 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet 8529a93abd6e4d62bbc0bddcf2fda78a with delete_type TABLET_DATA_TOMBSTONED (TS e7bcff2fef614e7a9981554c9e6db3c8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:59160
I20240502 01:59:33.475374 13778 consensus_queue.cc:1035] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20240502 01:59:33.476321 13786 tablet_replica.cc:330] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8: stopping tablet replica
I20240502 01:59:33.476521 13786 raft_consensus.cc:2229] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:33.476737 13786 raft_consensus.cc:2258] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:33.477312 13786 ts_tablet_manager.cc:1875] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:33.480136 13786 ts_tablet_manager.cc:1888] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:33.481011 13786 log.cc:1196] T a66b932d884e4df8a78841103d88ded0 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/a66b932d884e4df8a78841103d88ded0
I20240502 01:59:33.481622 6630 catalog_manager.cc:4830] TS e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): tablet a66b932d884e4df8a78841103d88ded0 (table rebalance_test_table_0 [id=6397b955acf741aeb8f869ab987d74b0]) successfully deleted
I20240502 01:59:33.481796 13786 tablet_replica.cc:330] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: stopping tablet replica
I20240502 01:59:33.481961 13786 raft_consensus.cc:2229] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:33.482085 13786 raft_consensus.cc:2258] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:33.482685 13786 ts_tablet_manager.cc:1875] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:33.484687 13786 ts_tablet_manager.cc:1888] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:33.484810 13786 log.cc:1196] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/8529a93abd6e4d62bbc0bddcf2fda78a
I20240502 01:59:33.485347 6630 catalog_manager.cc:4830] TS e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): tablet 8529a93abd6e4d62bbc0bddcf2fda78a (table rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]) successfully deleted
I20240502 01:59:33.525346 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:33.526176 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.528399 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.531559 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:33.543273 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.545384 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.553340 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:33.555274 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.563020 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:33.565176 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.566813 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:33.600754 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.621774 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:33.633113 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.646749 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.660032 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.692991 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.693616 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.699951 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.711484 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.719012 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:33.723920 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.724481 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:33.745573 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:33.792856 7181 consensus_queue.cc:237] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:33.794900 8757 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:33.794935 8508 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:33.795325 13584 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:33.795491 13630 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:33.796228 6783 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:33.796660 13630 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:33.802196 13584 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:33.802862 6783 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:33.804003 8757 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:33.807047 6639 catalog_manager.cc:5484] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 5 to 6, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:33.808910 8508 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:33.812031 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet e6ac26fa6de44f7ca68e6c0b6b959db7 with delete_type TABLET_DATA_TOMBSTONED (TS c6101231591a458384f4ea6d1f0e0ae5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36030
I20240502 01:59:33.815505 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.823503 13988 tablet_replica.cc:330] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5: stopping tablet replica
I20240502 01:59:33.823864 13988 raft_consensus.cc:2229] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:33.824250 13988 raft_consensus.cc:2258] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:33.825516 13988 ts_tablet_manager.cc:1875] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:33.828254 13988 ts_tablet_manager.cc:1888] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:59:33.828389 13988 log.cc:1196] T e6ac26fa6de44f7ca68e6c0b6b959db7 P c6101231591a458384f4ea6d1f0e0ae5: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/e6ac26fa6de44f7ca68e6c0b6b959db7
I20240502 01:59:33.830602 6630 catalog_manager.cc:4830] TS c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125): tablet e6ac26fa6de44f7ca68e6c0b6b959db7 (table rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]) successfully deleted
I20240502 01:59:33.832918 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.833637 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.834772 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.837081 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:33.838188 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:33.838594 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:33.876138 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:33.988049 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:33.991786 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.003882 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.005192 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.007344 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:34.019503 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.030560 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:34.063095 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.071746 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.077201 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.079468 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.080910 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:34.103617 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.123915 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:34.125116 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.144059 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:34.173861 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:34.174621 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:34.182013 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:34.185724 6917 consensus_queue.cc:237] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:34.186447 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.187708 8904 raft_consensus.cc:1261] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:34.188400 13679 consensus_queue.cc:1035] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:34.190809 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.203717 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.204381 13679 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:34.208634 8904 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:34.209419 6640 catalog_manager.cc:5484] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:34.215812 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.220017 7161 tablet_service.cc:1514] Processing DeleteTablet for tablet 39d9e48f67f34ebe850fc388ecd94ca3 with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36602
I20240502 01:59:34.226435 6917 consensus_queue.cc:237] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:34.229162 8904 raft_consensus.cc:1261] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:34.230469 7181 raft_consensus.cc:1261] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:34.232352 13679 consensus_queue.cc:1035] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:34.232678 13679 consensus_queue.cc:1035] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:34.241063 8904 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:34.241494 14230 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:34.242262 7181 raft_consensus.cc:858] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
W20240502 01:59:34.242329 7181 raft_consensus.cc:1565] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:34.243772 7181 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:34.244391 6640 catalog_manager.cc:5484] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d reported cstate change: config changed from index 6 to 7, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:34.249297 6785 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91db5c1b3d2f4d22b5882a61ea0395c7"
dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48284
I20240502 01:59:34.249318 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 39d9e48f67f34ebe850fc388ecd94ca3 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:59:34.249444 6785 raft_consensus.cc:595] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Received request to transfer leadership
I20240502 01:59:34.251351 7181 raft_consensus.cc:1467] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240502 01:59:34.251684 7181 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:34.266904 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.271210 13769 raft_consensus.cc:979] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e: : Instructing follower 86bc8001c3464b929cd5c873861baa0c to start an election
I20240502 01:59:34.271302 13769 raft_consensus.cc:1067] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Signalling peer 86bc8001c3464b929cd5c873861baa0c to start an election
I20240502 01:59:34.271956 7181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91db5c1b3d2f4d22b5882a61ea0395c7"
dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
from {username='slave'} at 127.27.111.193:43151
I20240502 01:59:34.272135 7181 raft_consensus.cc:482] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:34.272200 7181 raft_consensus.cc:3037] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:34.273278 7181 raft_consensus.cc:504] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:34.273794 7181 leader_election.cc:288] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 2 election: Requested vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
I20240502 01:59:34.274246 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91db5c1b3d2f4d22b5882a61ea0395c7" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:34.274351 6785 raft_consensus.cc:3032] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:34.274385 6785 raft_consensus.cc:729] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:34.274567 6785 consensus_queue.cc:260] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:34.274673 7048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91db5c1b3d2f4d22b5882a61ea0395c7" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:34.274741 6785 raft_consensus.cc:3037] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:34.274797 7048 raft_consensus.cc:3037] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:34.275745 6785 raft_consensus.cc:2454] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 2.
I20240502 01:59:34.275888 7048 raft_consensus.cc:2454] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 2.
I20240502 01:59:34.277531 7117 leader_election.cc:302] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, 86bc8001c3464b929cd5c873861baa0c, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:34.277719 12934 raft_consensus.cc:2781] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:34.277787 12934 raft_consensus.cc:686] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 2 LEADER]: Becoming Leader. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Running, Role: LEADER
I20240502 01:59:34.277916 12934 consensus_queue.cc:237] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:34.280797 6641 catalog_manager.cc:5484] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: term changed from 1 to 2, leader changed from 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) to 86bc8001c3464b929cd5c873861baa0c (127.27.111.196). New cstate: current_term: 2 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
W20240502 01:59:34.282841 6629 catalog_manager.cc:4804] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): delete failed for tablet 39d9e48f67f34ebe850fc388ecd94ca3 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 5 but the committed config has opid_index of 7
I20240502 01:59:34.315114 8641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91db5c1b3d2f4d22b5882a61ea0395c7" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4f53d80e39be410d92d4766692aff1c6"
I20240502 01:59:34.315354 8641 raft_consensus.cc:3037] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:34.316624 8641 raft_consensus.cc:2454] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 2.
I20240502 01:59:34.328372 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.329384 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:34.338723 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.351457 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.351555 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.351876 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.365306 6763 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:34.388536 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:34.427568 6917 consensus_queue.cc:237] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:34.429117 7181 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:34.429118 8904 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:59:34.429618 14233 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:34.429806 14230 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:34.431937 13679 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:34.432410 8904 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:34.432696 7182 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:34.435214 6636 catalog_manager.cc:5484] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 4 to 5, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:34.442605 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c55b199a24432e902cce9121fc0780 with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:34.447122 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.452173 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.460562 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:34.460995 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.463928 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.469756 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.494382 14411 tablet_replica.cc:330] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:34.494562 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:34.495618 6763 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:34.503430 8641 consensus_queue.cc:237] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:34.504226 14411 raft_consensus.cc:2229] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:34.504441 14411 raft_consensus.cc:2258] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:34.505064 14411 ts_tablet_manager.cc:1875] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:34.506722 7182 raft_consensus.cc:1261] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:34.507848 14411 ts_tablet_manager.cc:1888] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:59:34.508011 13778 consensus_queue.cc:1035] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:34.508595 14411 log.cc:1196] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/44c55b199a24432e902cce9121fc0780
I20240502 01:59:34.513648 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet 44c55b199a24432e902cce9121fc0780 (table rebalance_test_table_4 [id=f2b695d4b3e54fdfad81f019e9ab7b4a]) successfully deleted
I20240502 01:59:34.518596 13778 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:34.518944 7182 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:34.521624 6643 catalog_manager.cc:5484] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 5 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 2 leader_uuid: "4f53d80e39be410d92d4766692aff1c6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:34.525590 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet c8c237c50773494abf8813494be0f184 with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:34.527379 14411 tablet_replica.cc:330] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:34.527542 14411 raft_consensus.cc:2229] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:34.527666 14411 raft_consensus.cc:2258] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:34.529233 14411 ts_tablet_manager.cc:1875] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:34.531914 14411 ts_tablet_manager.cc:1888] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:59:34.532068 14411 log.cc:1196] T c8c237c50773494abf8813494be0f184 P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/c8c237c50773494abf8813494be0f184
I20240502 01:59:34.532338 8641 consensus_queue.cc:237] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:34.533418 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet c8c237c50773494abf8813494be0f184 (table rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]) successfully deleted
I20240502 01:59:34.536159 7182 raft_consensus.cc:1261] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:59:34.536572 14437 consensus_queue.cc:1035] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:34.540226 14437 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:34.540699 7182 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:34.542492 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c8c237c50773494abf8813494be0f184 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:59:34.543804 6636 catalog_manager.cc:5484] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 2 leader_uuid: "4f53d80e39be410d92d4766692aff1c6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
W20240502 01:59:34.544554 8574 consensus_peers.cc:485] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8c237c50773494abf8813494be0f184. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:34.674496 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.684041 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.686555 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:34.694957 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.708773 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:34.712038 6763 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:34.721182 8641 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:59:34.721892 12934 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:34.745172 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.749956 7050 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:59:34.750613 12934 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:34.755625 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.767735 6785 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:59:34.768185 13986 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:59:34.793303 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.804625 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.816380 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:34.818111 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.826916 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.832837 6763 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:34.848697 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.856585 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.858260 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:34.859928 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:34.860685 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.865520 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:34.875223 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.886302 6763 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:34.895046 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.905398 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.961767 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.978824 6763 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:34.979583 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.981084 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:34.982278 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.995461 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.020431 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:35.024446 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:35.045887 7050 consensus_queue.cc:237] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:35.047508 8641 raft_consensus.cc:1261] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:35.047907 12528 consensus_queue.cc:1035] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:35.050211 12528 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:35.052240 8641 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:35.054713 6642 catalog_manager.cc:5484] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:35.067552 7050 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "034cf1c3d47a4f7785c16d16815c229d"
dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42756
I20240502 01:59:35.067749 7050 raft_consensus.cc:595] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:59:35.067756 14865 ts_tablet_manager.cc:902] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
I20240502 01:59:35.068236 14865 tablet_copy_client.cc:308] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.198:46475
I20240502 01:59:35.078505 8662 tablet_copy_service.cc:140] P 4f53d80e39be410d92d4766692aff1c6: Received BeginTabletCopySession request for tablet c8c237c50773494abf8813494be0f184 from peer 070558c278e0457b9cc070ac0d662b5d ({username='slave'} at 127.27.111.200:51519)
I20240502 01:59:35.078673 8662 tablet_copy_service.cc:161] P 4f53d80e39be410d92d4766692aff1c6: Beginning new tablet copy session on tablet c8c237c50773494abf8813494be0f184 from peer 070558c278e0457b9cc070ac0d662b5d at {username='slave'} at 127.27.111.200:51519: session id = 070558c278e0457b9cc070ac0d662b5d-c8c237c50773494abf8813494be0f184
I20240502 01:59:35.080103 8662 tablet_copy_source_session.cc:205] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:35.081687 7050 consensus_queue.cc:237] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:35.083675 7161 tablet_service.cc:1514] Processing DeleteTablet for tablet 14e3c78c65334b52aca79d67c4bf45a2 with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36602
W20240502 01:59:35.083791 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): Couldn't send request to peer e7bcff2fef614e7a9981554c9e6db3c8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:35.086695 8641 raft_consensus.cc:1261] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:35.087569 14838 consensus_queue.cc:1035] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:35.087791 14885 tablet_replica.cc:330] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c: stopping tablet replica
I20240502 01:59:35.087992 14885 raft_consensus.cc:2229] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:35.088254 14885 raft_consensus.cc:2258] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:35.088514 6917 consensus_queue.cc:237] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:35.088940 14885 ts_tablet_manager.cc:1875] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:35.089531 12528 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 01:59:35.089938 8641 raft_consensus.cc:1261] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:35.090312 14233 consensus_queue.cc:1035] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:35.091851 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 14e3c78c65334b52aca79d67c4bf45a2 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:59:35.092036 14885 ts_tablet_manager.cc:1888] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:59:35.092156 14885 log.cc:1196] T 14e3c78c65334b52aca79d67c4bf45a2 P 86bc8001c3464b929cd5c873861baa0c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/14e3c78c65334b52aca79d67c4bf45a2
I20240502 01:59:35.092741 8642 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 01:59:35.092931 6629 catalog_manager.cc:4830] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): tablet 14e3c78c65334b52aca79d67c4bf45a2 (table rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]) successfully deleted
I20240502 01:59:35.095099 6634 catalog_manager.cc:5484] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 6 to 7, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 01:59:35.109848 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.112198 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.114380 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c237c50773494abf8813494be0f184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:35.118659 14865 tablet_copy_client.cc:783] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:35.118829 14865 tablet_copy_client.cc:647] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:35.119177 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:35.120311 14865 tablet_copy_client.cc:523] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:35.121661 14865 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap starting.
I20240502 01:59:35.124078 14233 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:35.124686 14838 raft_consensus.cc:979] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5: : Instructing follower 85e78945910a4e1abfa101d2edc3644e to start an election
I20240502 01:59:35.124763 14838 raft_consensus.cc:1067] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Signalling peer 85e78945910a4e1abfa101d2edc3644e to start an election
I20240502 01:59:35.125376 6785 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "034cf1c3d47a4f7785c16d16815c229d"
dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
from {username='slave'} at 127.27.111.195:55189
I20240502 01:59:35.125510 6785 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:35.125562 6785 raft_consensus.cc:3037] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:35.126684 6785 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:35.127135 6785 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 2 election: Requested vote from peers c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
I20240502 01:59:35.127919 6634 catalog_manager.cc:5484] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:35.128290 8641 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:35.129968 8642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034cf1c3d47a4f7785c16d16815c229d" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4f53d80e39be410d92d4766692aff1c6"
I20240502 01:59:35.130081 8642 raft_consensus.cc:3037] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:35.131110 8642 raft_consensus.cc:2454] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 2.
I20240502 01:59:35.131575 6721 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f53d80e39be410d92d4766692aff1c6, 85e78945910a4e1abfa101d2edc3644e; no voters:
I20240502 01:59:35.131844 7161 tablet_service.cc:1514] Processing DeleteTablet for tablet 818dbc9a8e054eeaa122e7919a311806 with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36602
I20240502 01:59:35.131940 14281 raft_consensus.cc:2781] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:35.132032 14281 raft_consensus.cc:686] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Becoming Leader. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:35.132103 14885 tablet_replica.cc:330] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c: stopping tablet replica
I20240502 01:59:35.132179 14281 consensus_queue.cc:237] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:35.132241 14885 raft_consensus.cc:2229] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:35.132395 14885 raft_consensus.cc:2258] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:35.132980 14885 ts_tablet_manager.cc:1875] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:35.133327 7050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034cf1c3d47a4f7785c16d16815c229d" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:35.133446 7050 raft_consensus.cc:3032] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:35.133486 7050 raft_consensus.cc:729] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Running, Role: LEADER
I20240502 01:59:35.133643 7050 consensus_queue.cc:260] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:35.133827 7050 raft_consensus.cc:3037] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:35.134929 7050 raft_consensus.cc:2454] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 2.
I20240502 01:59:35.135548 14885 ts_tablet_manager.cc:1888] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:59:35.135617 14885 log.cc:1196] T 818dbc9a8e054eeaa122e7919a311806 P 86bc8001c3464b929cd5c873861baa0c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/818dbc9a8e054eeaa122e7919a311806
I20240502 01:59:35.136163 6636 catalog_manager.cc:5484] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e reported cstate change: term changed from 1 to 2, leader changed from c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) to 85e78945910a4e1abfa101d2edc3644e (127.27.111.193). New cstate: current_term: 2 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:35.136332 6629 catalog_manager.cc:4830] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): tablet 818dbc9a8e054eeaa122e7919a311806 (table rebalance_test_table_4 [id=f2b695d4b3e54fdfad81f019e9ab7b4a]) successfully deleted
I20240502 01:59:35.136449 6917 consensus_queue.cc:237] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:35.140499 8642 raft_consensus.cc:1261] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:35.141503 7030 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:35.148670 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 818dbc9a8e054eeaa122e7919a311806. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:35.149001 14233 consensus_queue.cc:1035] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:35.152395 14233 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:35.154282 8642 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:35.154687 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.157179 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 818dbc9a8e054eeaa122e7919a311806 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:59:35.157415 6636 catalog_manager.cc:5484] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 6 to 7, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:35.162503 14865 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:35.162830 14865 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap complete.
I20240502 01:59:35.163017 14865 ts_tablet_manager.cc:1367] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d: Time spent bootstrapping tablet: real 0.041s user 0.030s sys 0.000s
I20240502 01:59:35.163456 14865 raft_consensus.cc:348] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:35.163570 14865 raft_consensus.cc:729] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Initialized, Role: FOLLOWER
I20240502 01:59:35.163717 14865 consensus_queue.cc:260] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:35.164145 14865 ts_tablet_manager.cc:1398] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:35.165831 8662 tablet_copy_service.cc:342] P 4f53d80e39be410d92d4766692aff1c6: Request end of tablet copy session 070558c278e0457b9cc070ac0d662b5d-c8c237c50773494abf8813494be0f184 received from {username='slave'} at 127.27.111.200:51519
I20240502 01:59:35.165936 8662 tablet_copy_service.cc:433] P 4f53d80e39be410d92d4766692aff1c6: ending tablet copy session 070558c278e0457b9cc070ac0d662b5d-c8c237c50773494abf8813494be0f184 on tablet c8c237c50773494abf8813494be0f184 with peer 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:35.179113 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.189121 7182 consensus_queue.cc:237] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:35.191012 7050 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:59:35.191910 14729 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:35.193576 8642 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:59:35.194056 14729 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:35.195060 14729 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:35.197322 7050 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:35.199725 6639 catalog_manager.cc:5484] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 4 to 6, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 2 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:35.200702 8642 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:35.203559 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 91db5c1b3d2f4d22b5882a61ea0395c7 with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:35.204010 14935 tablet_replica.cc:330] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:35.204176 14935 raft_consensus.cc:2229] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:35.204388 14935 raft_consensus.cc:2258] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:35.206517 14935 ts_tablet_manager.cc:1875] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:35.209820 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.210270 14935 ts_tablet_manager.cc:1888] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:59:35.210363 14935 log.cc:1196] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/91db5c1b3d2f4d22b5882a61ea0395c7
I20240502 01:59:35.210940 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet 91db5c1b3d2f4d22b5882a61ea0395c7 (table rebalance_test_table_9 [id=19f17ecb48ac4307bcd5af6811f21abe]) successfully deleted
I20240502 01:59:35.231081 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.285246 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.300107 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:35.300633 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.310385 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.323990 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.329476 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.352144 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.357235 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.378803 7030 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:35.378854 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.386272 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.398936 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.402081 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:35.407699 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.427649 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.430058 7030 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:35.454946 6916 consensus_queue.cc:237] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:35.456125 8904 raft_consensus.cc:1261] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:59:35.456604 14925 consensus_queue.cc:1035] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:35.458729 14917 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:35.461045 8904 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:35.461450 6641 catalog_manager.cc:5484] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 7 to 8, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:35.465291 7161 tablet_service.cc:1514] Processing DeleteTablet for tablet 39d9e48f67f34ebe850fc388ecd94ca3 with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:36602
I20240502 01:59:35.465516 14885 tablet_replica.cc:330] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c: stopping tablet replica
I20240502 01:59:35.465636 14885 raft_consensus.cc:2229] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:35.465737 14885 raft_consensus.cc:2258] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:35.466215 14885 ts_tablet_manager.cc:1875] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:35.466365 6916 consensus_queue.cc:237] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:35.468446 14885 ts_tablet_manager.cc:1888] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 01:59:35.468511 14885 log.cc:1196] T 39d9e48f67f34ebe850fc388ecd94ca3 P 86bc8001c3464b929cd5c873861baa0c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/39d9e48f67f34ebe850fc388ecd94ca3
W20240502 01:59:35.468899 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39d9e48f67f34ebe850fc388ecd94ca3. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:35.472746 6629 catalog_manager.cc:4830] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): tablet 39d9e48f67f34ebe850fc388ecd94ca3 (table rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]) successfully deleted
I20240502 01:59:35.478881 8904 raft_consensus.cc:1261] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:59:35.479314 14925 consensus_queue.cc:1035] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:35.481884 14917 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:35.483693 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 39d9e48f67f34ebe850fc388ecd94ca3 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:59:35.484313 8904 raft_consensus.cc:2932] T 39d9e48f67f34ebe850fc388ecd94ca3 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:35.484817 6639 catalog_manager.cc:5484] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 8 to 9, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:35.512205 15170 ts_tablet_manager.cc:902] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:35.513082 15170 tablet_copy_client.cc:308] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:35.513607 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 14e3c78c65334b52aca79d67c4bf45a2 from peer e7bcff2fef614e7a9981554c9e6db3c8 ({username='slave'} at 127.27.111.194:43103)
I20240502 01:59:35.513710 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 14e3c78c65334b52aca79d67c4bf45a2 from peer e7bcff2fef614e7a9981554c9e6db3c8 at {username='slave'} at 127.27.111.194:43103: session id = e7bcff2fef614e7a9981554c9e6db3c8-14e3c78c65334b52aca79d67c4bf45a2
I20240502 01:59:35.514765 7070 tablet_copy_source_session.cc:205] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:35.517009 15170 ts_tablet_manager.cc:1875] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:59:35.519765 15170 ts_tablet_manager.cc:1888] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 01:59:35.519973 15170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e3c78c65334b52aca79d67c4bf45a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:35.522107 15170 tablet_copy_client.cc:783] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:35.522280 15170 tablet_copy_client.cc:647] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:35.525125 15170 tablet_copy_client.cc:523] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:35.527415 15170 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 01:59:35.537848 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.544811 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.544817 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.547888 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.550485 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:35.563480 7030 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:35.565519 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.566226 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.618650 15211 ts_tablet_manager.cc:902] T 818dbc9a8e054eeaa122e7919a311806 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:35.625563 15211 tablet_copy_client.cc:308] T 818dbc9a8e054eeaa122e7919a311806 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 01:59:35.626426 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet 818dbc9a8e054eeaa122e7919a311806 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 ({username='slave'} at 127.27.111.199:52619)
I20240502 01:59:35.626526 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet 818dbc9a8e054eeaa122e7919a311806 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 at {username='slave'} at 127.27.111.199:52619: session id = 4e6e76b381af4cdc90bd3e9e6e8b5cf2-818dbc9a8e054eeaa122e7919a311806
I20240502 01:59:35.627437 6937 tablet_copy_source_session.cc:205] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:35.628610 15211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818dbc9a8e054eeaa122e7919a311806. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:35.630795 15211 tablet_copy_client.cc:783] T 818dbc9a8e054eeaa122e7919a311806 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:35.630949 15211 tablet_copy_client.cc:647] T 818dbc9a8e054eeaa122e7919a311806 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:35.633843 15211 tablet_copy_client.cc:523] T 818dbc9a8e054eeaa122e7919a311806 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:35.635653 15211 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap starting.
I20240502 01:59:35.637161 15170 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:35.637456 15170 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap complete.
I20240502 01:59:35.637620 15170 ts_tablet_manager.cc:1367] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.110s user 0.028s sys 0.003s
I20240502 01:59:35.638033 15170 raft_consensus.cc:348] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:35.638144 15170 raft_consensus.cc:729] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 01:59:35.638289 15170 consensus_queue.cc:260] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 01:59:35.638644 15170 ts_tablet_manager.cc:1398] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 01:59:35.639238 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session e7bcff2fef614e7a9981554c9e6db3c8-14e3c78c65334b52aca79d67c4bf45a2 received from {username='slave'} at 127.27.111.194:43103
I20240502 01:59:35.639309 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session e7bcff2fef614e7a9981554c9e6db3c8-14e3c78c65334b52aca79d67c4bf45a2 on tablet 14e3c78c65334b52aca79d67c4bf45a2 with peer e7bcff2fef614e7a9981554c9e6db3c8
I20240502 01:59:35.652556 8904 raft_consensus.cc:1203] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 01:59:35.684222 15211 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:35.684578 15211 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap complete.
I20240502 01:59:35.684767 15211 ts_tablet_manager.cc:1367] T 818dbc9a8e054eeaa122e7919a311806 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent bootstrapping tablet: real 0.049s user 0.027s sys 0.000s
I20240502 01:59:35.685343 8642 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:59:35.685900 14281 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:35.687981 15211 raft_consensus.cc:348] T 818dbc9a8e054eeaa122e7919a311806 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:35.688094 15211 raft_consensus.cc:729] T 818dbc9a8e054eeaa122e7919a311806 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, State: Initialized, Role: FOLLOWER
I20240502 01:59:35.688249 15211 consensus_queue.cc:260] T 818dbc9a8e054eeaa122e7919a311806 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:35.688745 15211 ts_tablet_manager.cc:1398] T 818dbc9a8e054eeaa122e7919a311806 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:59:35.689618 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-818dbc9a8e054eeaa122e7919a311806 received from {username='slave'} at 127.27.111.199:52619
I20240502 01:59:35.689704 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-818dbc9a8e054eeaa122e7919a311806 on tablet 818dbc9a8e054eeaa122e7919a311806 with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:35.694223 7050 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:59:35.694828 14281 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:35.731860 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.741034 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.742025 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.744783 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.754626 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.756733 7030 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:35.756831 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.761031 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.764851 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:35.767347 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.780843 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.791085 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.810571 7030 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:35.819782 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.828490 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:35.850492 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.937024 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.949155 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:35.963649 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.967391 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:35.976068 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.977340 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:35.984787 7030 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:35.988977 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.989233 7030 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:36.009043 15455 ts_tablet_manager.cc:902] T 39d9e48f67f34ebe850fc388ecd94ca3 P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:36.009635 15455 tablet_copy_client.cc:308] T 39d9e48f67f34ebe850fc388ecd94ca3 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 01:59:36.010288 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet 39d9e48f67f34ebe850fc388ecd94ca3 from peer 4f53d80e39be410d92d4766692aff1c6 ({username='slave'} at 127.27.111.198:35861)
I20240502 01:59:36.010452 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet 39d9e48f67f34ebe850fc388ecd94ca3 from peer 4f53d80e39be410d92d4766692aff1c6 at {username='slave'} at 127.27.111.198:35861: session id = 4f53d80e39be410d92d4766692aff1c6-39d9e48f67f34ebe850fc388ecd94ca3
I20240502 01:59:36.011528 6937 tablet_copy_source_session.cc:205] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:36.013773 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.015391 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.019732 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.023451 15455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d9e48f67f34ebe850fc388ecd94ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:36.028237 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.032004 15455 tablet_copy_client.cc:783] T 39d9e48f67f34ebe850fc388ecd94ca3 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:36.032231 15455 tablet_copy_client.cc:647] T 39d9e48f67f34ebe850fc388ecd94ca3 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:36.033664 15455 tablet_copy_client.cc:523] T 39d9e48f67f34ebe850fc388ecd94ca3 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:36.033686 8760 raft_consensus.cc:1203] T 818dbc9a8e054eeaa122e7919a311806 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:59:36.035207 15455 tablet_bootstrap.cc:492] T 39d9e48f67f34ebe850fc388ecd94ca3 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap starting.
I20240502 01:59:36.044719 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.045472 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.049450 6916 raft_consensus.cc:1203] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:59:36.060945 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.090242 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.098181 15455 tablet_bootstrap.cc:492] T 39d9e48f67f34ebe850fc388ecd94ca3 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:36.098541 15455 tablet_bootstrap.cc:492] T 39d9e48f67f34ebe850fc388ecd94ca3 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap complete.
I20240502 01:59:36.098870 15455 ts_tablet_manager.cc:1367] T 39d9e48f67f34ebe850fc388ecd94ca3 P 4f53d80e39be410d92d4766692aff1c6: Time spent bootstrapping tablet: real 0.064s user 0.021s sys 0.004s
I20240502 01:59:36.099345 15455 raft_consensus.cc:348] T 39d9e48f67f34ebe850fc388ecd94ca3 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:36.099455 15455 raft_consensus.cc:729] T 39d9e48f67f34ebe850fc388ecd94ca3 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Initialized, Role: FOLLOWER
I20240502 01:59:36.099623 15455 consensus_queue.cc:260] T 39d9e48f67f34ebe850fc388ecd94ca3 P 4f53d80e39be410d92d4766692aff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:36.100013 15455 ts_tablet_manager.cc:1398] T 39d9e48f67f34ebe850fc388ecd94ca3 P 4f53d80e39be410d92d4766692aff1c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:36.100633 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 4f53d80e39be410d92d4766692aff1c6-39d9e48f67f34ebe850fc388ecd94ca3 received from {username='slave'} at 127.27.111.198:35861
I20240502 01:59:36.100726 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 4f53d80e39be410d92d4766692aff1c6-39d9e48f67f34ebe850fc388ecd94ca3 on tablet 39d9e48f67f34ebe850fc388ecd94ca3 with peer 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:36.115160 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.124666 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.126739 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:36.141666 7030 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:36.151286 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.151836 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.183816 6785 consensus_queue.cc:237] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:36.184496 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.186075 8642 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:36.186621 14281 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:36.188984 14281 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:36.191586 8642 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:36.191843 6642 catalog_manager.cc:5484] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 5 to 7, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New cstate: current_term: 2 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:36.198077 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet 034cf1c3d47a4f7785c16d16815c229d with delete_type TABLET_DATA_TOMBSTONED (TS c6101231591a458384f4ea6d1f0e0ae5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36030
I20240502 01:59:36.198468 15490 tablet_replica.cc:330] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5: stopping tablet replica
I20240502 01:59:36.198622 15490 raft_consensus.cc:2229] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:36.198957 15490 raft_consensus.cc:2258] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:36.199170 6785 consensus_queue.cc:237] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:36.201242 8642 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:59:36.201723 14281 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:59:36.202416 6718 consensus_peers.cc:485] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 034cf1c3d47a4f7785c16d16815c229d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:36.203783 15302 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:36.204173 8642 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:36.205515 6629 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 034cf1c3d47a4f7785c16d16815c229d with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:59:36.206025 15490 ts_tablet_manager.cc:1875] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:36.206626 6637 catalog_manager.cc:5484] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 2 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:36.209689 15490 ts_tablet_manager.cc:1888] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:59:36.209767 15490 log.cc:1196] T 034cf1c3d47a4f7785c16d16815c229d P c6101231591a458384f4ea6d1f0e0ae5: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/034cf1c3d47a4f7785c16d16815c229d
I20240502 01:59:36.210269 6630 catalog_manager.cc:4830] TS c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125): tablet 034cf1c3d47a4f7785c16d16815c229d (table rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]) successfully deleted
I20240502 01:59:36.261288 7182 consensus_queue.cc:237] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:36.263202 8903 raft_consensus.cc:1261] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:36.264170 12934 consensus_queue.cc:1035] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:36.269551 7050 consensus_queue.cc:237] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:36.271132 8642 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:36.271708 14887 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:36.274569 12934 raft_consensus.cc:2932] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:36.277665 8903 raft_consensus.cc:2932] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:36.280105 6637 catalog_manager.cc:5484] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 5 to 6, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:36.282761 15491 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:36.283280 8642 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:36.285533 7182 consensus_queue.cc:237] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:36.285728 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet 4154be3f1f3c428580463c419d69892e with delete_type TABLET_DATA_TOMBSTONED (TS c6101231591a458384f4ea6d1f0e0ae5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36030
I20240502 01:59:36.286672 15490 tablet_replica.cc:330] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5: stopping tablet replica
I20240502 01:59:36.286969 15490 raft_consensus.cc:2229] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:36.287029 6636 catalog_manager.cc:5484] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:36.287137 15490 raft_consensus.cc:2258] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:36.287276 8903 raft_consensus.cc:1261] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:59:36.287464 7116 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): Couldn't send request to peer 85e78945910a4e1abfa101d2edc3644e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:36.287762 15490 ts_tablet_manager.cc:1875] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:36.287979 12934 consensus_queue.cc:1035] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:36.289708 12934 raft_consensus.cc:2932] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:36.291752 8903 raft_consensus.cc:2932] T 4154be3f1f3c428580463c419d69892e P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:36.293377 7050 consensus_queue.cc:237] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:36.293927 15490 ts_tablet_manager.cc:1888] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:59:36.294062 15490 log.cc:1196] T 4154be3f1f3c428580463c419d69892e P c6101231591a458384f4ea6d1f0e0ae5: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/4154be3f1f3c428580463c419d69892e
I20240502 01:59:36.294715 6630 catalog_manager.cc:4830] TS c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125): tablet 4154be3f1f3c428580463c419d69892e (table rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]) successfully deleted
I20240502 01:59:36.295260 8642 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:36.295243 6634 catalog_manager.cc:5484] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 6 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
W20240502 01:59:36.295516 6982 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47a276e9780e448a9b3c6eab67925514. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:36.295660 14887 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:36.297187 15491 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:36.297897 6629 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4154be3f1f3c428580463c419d69892e with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:59:36.298228 7161 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a276e9780e448a9b3c6eab67925514 with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36602
I20240502 01:59:36.298835 8642 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:36.300572 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 47a276e9780e448a9b3c6eab67925514 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:59:36.301692 6639 catalog_manager.cc:5484] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 6 to 7, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:36.308671 15591 tablet_replica.cc:330] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: stopping tablet replica
I20240502 01:59:36.308869 15591 raft_consensus.cc:2229] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:36.309069 15591 raft_consensus.cc:2258] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:36.314056 15591 ts_tablet_manager.cc:1875] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:36.318264 15591 ts_tablet_manager.cc:1888] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:59:36.318375 15591 log.cc:1196] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/47a276e9780e448a9b3c6eab67925514
I20240502 01:59:36.319013 6629 catalog_manager.cc:4830] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): tablet 47a276e9780e448a9b3c6eab67925514 (table rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]) successfully deleted
I20240502 01:59:36.326344 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.328002 6785 consensus_queue.cc:237] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:36.330689 8760 raft_consensus.cc:1261] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:36.331391 6916 raft_consensus.cc:1261] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 01:59:36.331648 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4fb9af11abc49d496b5c3d8048c22c8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:36.331902 15302 consensus_queue.cc:1035] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:36.332486 15302 consensus_queue.cc:1035] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:36.338003 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:36.342840 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.346769 15302 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:36.350039 6916 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:36.350093 8760 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:36.351239 6637 catalog_manager.cc:5484] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:36.364086 7030 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:36.369843 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.388641 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:36.394078 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.398550 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.398808 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.403585 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.420672 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.422765 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.428252 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.453138 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.454151 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.481602 7030 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:36.503185 8642 raft_consensus.cc:1203] T 39d9e48f67f34ebe850fc388ecd94ca3 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 01:59:36.519348 6916 consensus_queue.cc:237] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:36.519587 7182 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7"
dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38188
I20240502 01:59:36.519752 7182 raft_consensus.cc:595] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Received request to transfer leadership
I20240502 01:59:36.521787 7050 raft_consensus.cc:1261] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:36.522270 14925 consensus_queue.cc:1035] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:59:36.522504 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcc5daef41bc44c4855f2bcb44e1c6d5. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:36.523100 8760 raft_consensus.cc:1261] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:36.523769 14925 consensus_queue.cc:1035] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:36.525487 7182 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91db5c1b3d2f4d22b5882a61ea0395c7"
dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38176
I20240502 01:59:36.525604 7182 raft_consensus.cc:595] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 2 LEADER]: Received request to transfer leadership
I20240502 01:59:36.525745 15704 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:36.527802 8760 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:36.531280 6642 catalog_manager.cc:5484] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 reported cstate change: config changed from index 5 to 6, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:36.531939 7050 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:36.570020 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.571803 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.586061 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:36.594345 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.595196 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.595810 7030 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:36.607935 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.613296 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.642823 15585 raft_consensus.cc:979] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: : Instructing follower 85e78945910a4e1abfa101d2edc3644e to start an election
I20240502 01:59:36.642940 15585 raft_consensus.cc:1067] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Signalling peer 85e78945910a4e1abfa101d2edc3644e to start an election
I20240502 01:59:36.643640 6785 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7"
dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
from {username='slave'} at 127.27.111.196:45315
I20240502 01:59:36.643779 6785 raft_consensus.cc:482] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:36.643846 6785 raft_consensus.cc:3037] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:36.645045 6785 raft_consensus.cc:504] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:36.645567 6785 leader_election.cc:288] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 2 election: Requested vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741), 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767)
I20240502 01:59:36.645831 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:36.645948 7182 raft_consensus.cc:3032] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:36.645992 7182 raft_consensus.cc:729] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Running, Role: LEADER
I20240502 01:59:36.646153 8509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "954cd49326bf405abd81182d75dc783c"
I20240502 01:59:36.646155 7182 consensus_queue.cc:260] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:36.646272 8760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7" candidate_uuid: "85e78945910a4e1abfa101d2edc3644e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2"
I20240502 01:59:36.646348 7182 raft_consensus.cc:3037] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:36.646345 8509 raft_consensus.cc:3037] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:36.646484 8760 raft_consensus.cc:3037] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:36.647454 7182 raft_consensus.cc:2454] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 2.
I20240502 01:59:36.647460 8509 raft_consensus.cc:2454] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 2.
I20240502 01:59:36.647722 8760 raft_consensus.cc:2454] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85e78945910a4e1abfa101d2edc3644e in term 2.
I20240502 01:59:36.648458 6720 leader_election.cc:302] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 85e78945910a4e1abfa101d2edc3644e, 86bc8001c3464b929cd5c873861baa0c, 954cd49326bf405abd81182d75dc783c; no voters:
I20240502 01:59:36.648634 15644 raft_consensus.cc:2781] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:36.648730 15644 raft_consensus.cc:686] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Becoming Leader. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:36.648862 15644 consensus_queue.cc:237] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:36.651372 6638 catalog_manager.cc:5484] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: term changed from 1 to 2, leader changed from 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) to 85e78945910a4e1abfa101d2edc3644e (127.27.111.193). New cstate: current_term: 2 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:36.654997 12934 raft_consensus.cc:979] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c: : Instructing follower c6101231591a458384f4ea6d1f0e0ae5 to start an election
I20240502 01:59:36.655088 12934 raft_consensus.cc:1067] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 2 LEADER]: Signalling peer c6101231591a458384f4ea6d1f0e0ae5 to start an election
I20240502 01:59:36.656715 7050 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91db5c1b3d2f4d22b5882a61ea0395c7"
dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
from {username='slave'} at 127.27.111.196:45961
I20240502 01:59:36.656877 7050 raft_consensus.cc:482] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:36.656960 7050 raft_consensus.cc:3037] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:36.657898 7050 raft_consensus.cc:504] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:36.658404 7050 leader_election.cc:288] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 3 election: Requested vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
I20240502 01:59:36.658756 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91db5c1b3d2f4d22b5882a61ea0395c7" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:36.658887 7182 raft_consensus.cc:3032] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:59:36.658938 7182 raft_consensus.cc:729] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Running, Role: LEADER
I20240502 01:59:36.658969 8642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91db5c1b3d2f4d22b5882a61ea0395c7" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: true dest_uuid: "4f53d80e39be410d92d4766692aff1c6"
I20240502 01:59:36.659104 8642 raft_consensus.cc:3037] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:36.659113 7182 consensus_queue.cc:260] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:36.659273 7182 raft_consensus.cc:3037] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:36.660189 8642 raft_consensus.cc:2454] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 3.
I20240502 01:59:36.660346 7182 raft_consensus.cc:2454] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 3.
I20240502 01:59:36.660529 6985 leader_election.cc:302] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f53d80e39be410d92d4766692aff1c6, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:36.660727 14887 raft_consensus.cc:2781] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:59:36.660820 14887 raft_consensus.cc:686] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 3 LEADER]: Becoming Leader. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Running, Role: LEADER
I20240502 01:59:36.660965 14887 consensus_queue.cc:237] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:36.662917 6643 catalog_manager.cc:5484] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: term changed from 2 to 3, leader changed from 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) to c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195). New cstate: current_term: 3 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:36.693152 8642 consensus_queue.cc:237] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:36.695448 8903 raft_consensus.cc:1261] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:59:36.696179 7182 raft_consensus.cc:1261] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:59:36.696774 13778 consensus_queue.cc:1035] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:36.698099 13778 consensus_queue.cc:1035] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:36.704099 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.707425 13778 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:36.710054 8903 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:36.710373 6637 catalog_manager.cc:5484] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 8 to 9, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 2 leader_uuid: "4f53d80e39be410d92d4766692aff1c6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:36.711910 7182 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:36.726348 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:36.728848 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.729169 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:36.743845 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:36.752416 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.764955 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.766440 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:36.768213 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:36.773340 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:36.779798 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.788249 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.791697 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:36.802290 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:36.807683 8574 consensus_peers.cc:485] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8c237c50773494abf8813494be0f184. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:36.809846 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.820839 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.822751 8621 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:36.828749 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.833159 6785 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a66b932d884e4df8a78841103d88ded0"
dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48652
I20240502 01:59:36.833321 6785 raft_consensus.cc:595] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Received request to transfer leadership
I20240502 01:59:36.841805 15861 ts_tablet_manager.cc:902] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:36.842867 15861 tablet_copy_client.cc:308] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:36569
I20240502 01:59:36.843515 7201 tablet_copy_service.cc:140] P 86bc8001c3464b929cd5c873861baa0c: Received BeginTabletCopySession request for tablet 4154be3f1f3c428580463c419d69892e from peer 85e78945910a4e1abfa101d2edc3644e ({username='slave'} at 127.27.111.193:43151)
I20240502 01:59:36.843627 7201 tablet_copy_service.cc:161] P 86bc8001c3464b929cd5c873861baa0c: Beginning new tablet copy session on tablet 4154be3f1f3c428580463c419d69892e from peer 85e78945910a4e1abfa101d2edc3644e at {username='slave'} at 127.27.111.193:43151: session id = 85e78945910a4e1abfa101d2edc3644e-4154be3f1f3c428580463c419d69892e
I20240502 01:59:36.843910 6763 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:36.844604 7201 tablet_copy_source_session.cc:205] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:36.846828 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:36.851713 15497 raft_consensus.cc:979] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e: : Instructing follower 4f53d80e39be410d92d4766692aff1c6 to start an election
I20240502 01:59:36.851801 15497 raft_consensus.cc:1067] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Signalling peer 4f53d80e39be410d92d4766692aff1c6 to start an election
I20240502 01:59:36.852201 8642 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a66b932d884e4df8a78841103d88ded0"
dest_uuid: "4f53d80e39be410d92d4766692aff1c6"
from {username='slave'} at 127.27.111.193:36281
I20240502 01:59:36.852344 8642 raft_consensus.cc:482] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:36.852409 8642 raft_consensus.cc:3037] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:36.853469 8642 raft_consensus.cc:504] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:36.853884 8642 leader_election.cc:288] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [CANDIDATE]: Term 2 election: Requested vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:36.854568 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66b932d884e4df8a78841103d88ded0" candidate_uuid: "4f53d80e39be410d92d4766692aff1c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:36.854570 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66b932d884e4df8a78841103d88ded0" candidate_uuid: "4f53d80e39be410d92d4766692aff1c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
I20240502 01:59:36.854702 7182 raft_consensus.cc:3037] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:36.854701 6785 raft_consensus.cc:3032] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:36.854753 6785 raft_consensus.cc:729] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:36.854912 6785 consensus_queue.cc:260] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:36.855079 6785 raft_consensus.cc:3037] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:36.855765 7182 raft_consensus.cc:2454] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f53d80e39be410d92d4766692aff1c6 in term 2.
I20240502 01:59:36.856081 8576 leader_election.cc:302] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f53d80e39be410d92d4766692aff1c6, 86bc8001c3464b929cd5c873861baa0c; no voters:
I20240502 01:59:36.856180 6785 raft_consensus.cc:2454] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f53d80e39be410d92d4766692aff1c6 in term 2.
I20240502 01:59:36.858024 15839 raft_consensus.cc:2781] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:36.858183 15839 raft_consensus.cc:686] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [term 2 LEADER]: Becoming Leader. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Running, Role: LEADER
I20240502 01:59:36.859092 15839 consensus_queue.cc:237] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:36.861255 6638 catalog_manager.cc:5484] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: term changed from 1 to 2, leader changed from 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) to 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198). New cstate: current_term: 2 leader_uuid: "4f53d80e39be410d92d4766692aff1c6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:36.863827 15882 ts_tablet_manager.cc:902] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:36.863936 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.864392 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:36.864636 15861 ts_tablet_manager.cc:1875] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:59:36.864845 15882 tablet_copy_client.cc:308] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:36.865301 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet a4fb9af11abc49d496b5c3d8048c22c8 from peer 4f53d80e39be410d92d4766692aff1c6 ({username='slave'} at 127.27.111.198:60983)
I20240502 01:59:36.865392 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet a4fb9af11abc49d496b5c3d8048c22c8 from peer 4f53d80e39be410d92d4766692aff1c6 at {username='slave'} at 127.27.111.198:60983: session id = 4f53d80e39be410d92d4766692aff1c6-a4fb9af11abc49d496b5c3d8048c22c8
I20240502 01:59:36.866284 6805 tablet_copy_source_session.cc:205] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:36.867142 15861 ts_tablet_manager.cc:1888] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 01:59:36.867316 15861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4154be3f1f3c428580463c419d69892e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:36.867312 15882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4fb9af11abc49d496b5c3d8048c22c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:36.868474 15861 tablet_copy_client.cc:783] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:36.868661 15861 tablet_copy_client.cc:647] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:36.869915 15882 tablet_copy_client.cc:783] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:36.870206 15882 tablet_copy_client.cc:647] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:36.871762 15861 tablet_copy_client.cc:523] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:36.873543 15861 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:36.880187 15882 tablet_copy_client.cc:523] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:36.883255 15882 tablet_bootstrap.cc:492] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap starting.
I20240502 01:59:36.892411 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:36.898972 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:36.913189 15880 ts_tablet_manager.cc:902] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:36.913820 15880 tablet_copy_client.cc:308] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:36.914237 15881 ts_tablet_manager.cc:902] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:36.914419 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet 034cf1c3d47a4f7785c16d16815c229d from peer 070558c278e0457b9cc070ac0d662b5d ({username='slave'} at 127.27.111.200:36335)
I20240502 01:59:36.914546 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet 034cf1c3d47a4f7785c16d16815c229d from peer 070558c278e0457b9cc070ac0d662b5d at {username='slave'} at 127.27.111.200:36335: session id = 070558c278e0457b9cc070ac0d662b5d-034cf1c3d47a4f7785c16d16815c229d
I20240502 01:59:36.914794 15881 tablet_copy_client.cc:308] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:36.915254 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 47a276e9780e448a9b3c6eab67925514 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 ({username='slave'} at 127.27.111.199:49969)
I20240502 01:59:36.915362 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 47a276e9780e448a9b3c6eab67925514 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 at {username='slave'} at 127.27.111.199:49969: session id = 4e6e76b381af4cdc90bd3e9e6e8b5cf2-47a276e9780e448a9b3c6eab67925514
I20240502 01:59:36.915494 6805 tablet_copy_source_session.cc:205] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:36.916246 7070 tablet_copy_source_session.cc:205] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:36.917286 15880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034cf1c3d47a4f7785c16d16815c229d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:36.919072 15881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a276e9780e448a9b3c6eab67925514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:36.926046 15881 tablet_copy_client.cc:783] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:36.926236 15881 tablet_copy_client.cc:647] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:36.936870 15880 tablet_copy_client.cc:783] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:36.937160 15880 tablet_copy_client.cc:647] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:36.938385 15880 tablet_copy_client.cc:523] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:36.940093 15880 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d: Bootstrap starting.
I20240502 01:59:36.941118 15881 tablet_copy_client.cc:523] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:36.946192 15881 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap starting.
I20240502 01:59:36.963201 15882 tablet_bootstrap.cc:492] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:36.963574 15882 tablet_bootstrap.cc:492] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap complete.
I20240502 01:59:36.963807 15882 ts_tablet_manager.cc:1367] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6: Time spent bootstrapping tablet: real 0.081s user 0.021s sys 0.007s
I20240502 01:59:36.964336 15882 raft_consensus.cc:348] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:36.964504 15882 raft_consensus.cc:729] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Initialized, Role: FOLLOWER
I20240502 01:59:36.964659 15882 consensus_queue.cc:260] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:36.965770 15882 ts_tablet_manager.cc:1398] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 01:59:36.967399 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 4f53d80e39be410d92d4766692aff1c6-a4fb9af11abc49d496b5c3d8048c22c8 received from {username='slave'} at 127.27.111.198:60983
I20240502 01:59:36.967483 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 4f53d80e39be410d92d4766692aff1c6-a4fb9af11abc49d496b5c3d8048c22c8 on tablet a4fb9af11abc49d496b5c3d8048c22c8 with peer 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:36.970715 15861 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:36.971056 15861 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: Bootstrap complete.
I20240502 01:59:36.971232 15861 ts_tablet_manager.cc:1367] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.098s user 0.026s sys 0.001s
I20240502 01:59:36.971635 15861 raft_consensus.cc:348] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:36.971750 15861 raft_consensus.cc:729] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:36.971916 15861 consensus_queue.cc:260] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:36.972613 15861 ts_tablet_manager.cc:1398] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:36.973228 7201 tablet_copy_service.cc:342] P 86bc8001c3464b929cd5c873861baa0c: Request end of tablet copy session 85e78945910a4e1abfa101d2edc3644e-4154be3f1f3c428580463c419d69892e received from {username='slave'} at 127.27.111.193:43151
I20240502 01:59:36.973309 7201 tablet_copy_service.cc:433] P 86bc8001c3464b929cd5c873861baa0c: ending tablet copy session 85e78945910a4e1abfa101d2edc3644e-4154be3f1f3c428580463c419d69892e on tablet 4154be3f1f3c428580463c419d69892e with peer 85e78945910a4e1abfa101d2edc3644e
I20240502 01:59:37.010136 15880 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:37.010848 15880 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d: Bootstrap complete.
I20240502 01:59:37.011673 15880 ts_tablet_manager.cc:1367] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d: Time spent bootstrapping tablet: real 0.072s user 0.029s sys 0.000s
I20240502 01:59:37.013476 15880 raft_consensus.cc:348] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:37.013685 15880 raft_consensus.cc:729] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Initialized, Role: FOLLOWER
I20240502 01:59:37.013928 15880 consensus_queue.cc:260] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:37.015439 15880 ts_tablet_manager.cc:1398] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:37.016487 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 070558c278e0457b9cc070ac0d662b5d-034cf1c3d47a4f7785c16d16815c229d received from {username='slave'} at 127.27.111.200:36335
I20240502 01:59:37.016562 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 070558c278e0457b9cc070ac0d662b5d-034cf1c3d47a4f7785c16d16815c229d on tablet 034cf1c3d47a4f7785c16d16815c229d with peer 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:37.025031 15881 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:37.025374 15881 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap complete.
I20240502 01:59:37.025584 15881 ts_tablet_manager.cc:1367] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent bootstrapping tablet: real 0.079s user 0.027s sys 0.000s
I20240502 01:59:37.026122 15881 raft_consensus.cc:348] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:37.026273 15881 raft_consensus.cc:729] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, State: Initialized, Role: FOLLOWER
I20240502 01:59:37.026439 15881 consensus_queue.cc:260] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:37.026796 15881 ts_tablet_manager.cc:1398] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 01:59:37.027483 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-47a276e9780e448a9b3c6eab67925514 received from {username='slave'} at 127.27.111.199:49969
I20240502 01:59:37.027581 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-47a276e9780e448a9b3c6eab67925514 on tablet 47a276e9780e448a9b3c6eab67925514 with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:37.048132 15880 ts_tablet_manager.cc:902] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:37.048777 15880 tablet_copy_client.cc:308] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 01:59:37.049257 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet dcc5daef41bc44c4855f2bcb44e1c6d5 from peer 070558c278e0457b9cc070ac0d662b5d ({username='slave'} at 127.27.111.200:54661)
I20240502 01:59:37.049363 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet dcc5daef41bc44c4855f2bcb44e1c6d5 from peer 070558c278e0457b9cc070ac0d662b5d at {username='slave'} at 127.27.111.200:54661: session id = 070558c278e0457b9cc070ac0d662b5d-dcc5daef41bc44c4855f2bcb44e1c6d5
I20240502 01:59:37.050381 6937 tablet_copy_source_session.cc:205] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:37.051332 15880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc5daef41bc44c4855f2bcb44e1c6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:37.053900 15880 tablet_copy_client.cc:783] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:37.054068 15880 tablet_copy_client.cc:647] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:37.056784 15880 tablet_copy_client.cc:523] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:37.059556 15880 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap starting.
I20240502 01:59:37.086679 8509 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:37.087157 15497 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:37.106943 8760 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:37.107368 15497 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:37.112540 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.117913 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:37.118091 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.118780 15881 ts_tablet_manager.cc:902] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
I20240502 01:59:37.121228 15881 tablet_copy_client.cc:308] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.198:46475
I20240502 01:59:37.127015 7182 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:37.127594 15497 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:37.130810 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:37.131704 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:37.132593 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.138975 8642 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 01:59:37.139411 14887 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:37.139591 6785 raft_consensus.cc:1203] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:59:37.152920 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:37.170195 7181 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 3 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 01:59:37.170738 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:37.171217 8903 raft_consensus.cc:1203] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 01:59:37.173431 14887 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:37.184708 8662 tablet_copy_service.cc:140] P 4f53d80e39be410d92d4766692aff1c6: Received BeginTabletCopySession request for tablet c8c237c50773494abf8813494be0f184 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 ({username='slave'} at 127.27.111.199:39193)
I20240502 01:59:37.184902 8662 tablet_copy_service.cc:161] P 4f53d80e39be410d92d4766692aff1c6: Beginning new tablet copy session on tablet c8c237c50773494abf8813494be0f184 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 at {username='slave'} at 127.27.111.199:39193: session id = 4e6e76b381af4cdc90bd3e9e6e8b5cf2-c8c237c50773494abf8813494be0f184
I20240502 01:59:37.186061 8662 tablet_copy_source_session.cc:205] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:37.187183 15881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c237c50773494abf8813494be0f184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:37.188613 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.189386 15881 tablet_copy_client.cc:783] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:37.189524 15881 tablet_copy_client.cc:647] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:37.190762 15881 tablet_copy_client.cc:523] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:37.191942 15881 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap starting.
I20240502 01:59:37.200033 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.202432 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:37.213335 15880 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:37.214547 15880 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap complete.
I20240502 01:59:37.214764 15880 ts_tablet_manager.cc:1367] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d: Time spent bootstrapping tablet: real 0.155s user 0.020s sys 0.010s
I20240502 01:59:37.215237 15880 raft_consensus.cc:348] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:37.216697 15880 raft_consensus.cc:729] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Initialized, Role: FOLLOWER
I20240502 01:59:37.216878 15880 consensus_queue.cc:260] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:37.217260 15880 ts_tablet_manager.cc:1398] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:37.221199 8641 raft_consensus.cc:1203] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:59:37.223595 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:37.227821 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 070558c278e0457b9cc070ac0d662b5d-dcc5daef41bc44c4855f2bcb44e1c6d5 received from {username='slave'} at 127.27.111.200:54661
I20240502 01:59:37.227913 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 070558c278e0457b9cc070ac0d662b5d-dcc5daef41bc44c4855f2bcb44e1c6d5 on tablet dcc5daef41bc44c4855f2bcb44e1c6d5 with peer 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:37.236778 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.255333 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:37.265175 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:37.268465 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:37.295163 6785 raft_consensus.cc:1261] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:59:37.295768 15840 consensus_queue.cc:1035] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:37.299419 7181 raft_consensus.cc:1261] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:59:37.300740 15839 consensus_queue.cc:1035] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:37.317840 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.322876 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:37.328394 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.333366 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:37.359107 8760 raft_consensus.cc:1203] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:59:37.361812 15881 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:37.362130 15881 tablet_bootstrap.cc:492] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap complete.
I20240502 01:59:37.362330 15881 ts_tablet_manager.cc:1367] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent bootstrapping tablet: real 0.170s user 0.027s sys 0.004s
I20240502 01:59:37.362881 15881 raft_consensus.cc:348] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:37.363021 15881 raft_consensus.cc:729] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, State: Initialized, Role: FOLLOWER
I20240502 01:59:37.363179 15881 consensus_queue.cc:260] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:37.363472 15881 ts_tablet_manager.cc:1398] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:37.363937 8662 tablet_copy_service.cc:342] P 4f53d80e39be410d92d4766692aff1c6: Request end of tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-c8c237c50773494abf8813494be0f184 received from {username='slave'} at 127.27.111.199:39193
I20240502 01:59:37.364037 8662 tablet_copy_service.cc:433] P 4f53d80e39be410d92d4766692aff1c6: ending tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-c8c237c50773494abf8813494be0f184 on tablet c8c237c50773494abf8813494be0f184 with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:37.371457 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:37.378607 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:37.382711 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:37.390200 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:37.393688 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.396147 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.404429 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.413164 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:37.424533 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:37.440441 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:37.444620 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.465660 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:37.506080 6916 consensus_queue.cc:237] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:37.507879 8903 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:37.507933 7181 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:37.508364 15720 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:59:37.508654 6852 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44c55b199a24432e902cce9121fc0780. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:37.508729 15720 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:37.528314 16231 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:37.530655 7181 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:37.533528 6637 catalog_manager.cc:5484] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 5 to 6, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:37.538340 8903 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:37.543534 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.545240 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.572798 6785 consensus_queue.cc:237] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:37.575016 8760 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:59:37.575491 8509 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:59:37.575498 15497 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:37.575971 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:37.577236 15497 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:37.579015 15644 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:37.579793 8760 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:37.580195 8509 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:37.580317 7050 consensus_queue.cc:237] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:37.583413 6635 catalog_manager.cc:5484] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c reported cstate change: config changed from index 6 to 8, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 2 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:37.584481 8642 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 01:59:37.585167 16110 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:37.586843 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:37.591184 16105 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:37.591771 8642 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 6 to 8, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:37.593863 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.594839 6643 catalog_manager.cc:5484] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 6 to 8, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) evicted. New cstate: current_term: 3 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:37.603049 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:37.609084 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.612264 8903 raft_consensus.cc:1203] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:59:37.621145 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:37.624958 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:37.633092 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:37.638515 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:37.640830 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:37.642241 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.662560 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:37.679262 7159 tablet_service.cc:1514] Processing DeleteTablet for tablet 91db5c1b3d2f4d22b5882a61ea0395c7 with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:36602
I20240502 01:59:37.679530 7161 tablet_service.cc:1514] Processing DeleteTablet for tablet e6ac26fa6de44f7ca68e6c0b6b959db7 with delete_type TABLET_DATA_TOMBSTONED (TS 86bc8001c3464b929cd5c873861baa0c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:36602
I20240502 01:59:37.682142 7050 consensus_queue.cc:237] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:37.683751 8757 raft_consensus.cc:1203] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20240502 01:59:37.683893 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91db5c1b3d2f4d22b5882a61ea0395c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:37.690761 8642 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 01:59:37.691064 16324 tablet_replica.cc:330] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c: stopping tablet replica
I20240502 01:59:37.691234 16324 raft_consensus.cc:2229] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:37.691390 16324 raft_consensus.cc:2258] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:37.692099 16110 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:37.692184 16324 ts_tablet_manager.cc:1875] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:37.694262 16110 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:37.695144 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:37.695508 16324 ts_tablet_manager.cc:1888] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.7
I20240502 01:59:37.695600 16324 log.cc:1196] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 86bc8001c3464b929cd5c873861baa0c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/91db5c1b3d2f4d22b5882a61ea0395c7
I20240502 01:59:37.695991 8642 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:37.696216 6629 catalog_manager.cc:4830] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): tablet 91db5c1b3d2f4d22b5882a61ea0395c7 (table rebalance_test_table_9 [id=19f17ecb48ac4307bcd5af6811f21abe]) successfully deleted
I20240502 01:59:37.697939 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 91db5c1b3d2f4d22b5882a61ea0395c7 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:59:37.698833 6641 catalog_manager.cc:5484] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 8 to 9, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 3 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:37.699054 16324 tablet_replica.cc:330] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: stopping tablet replica
I20240502 01:59:37.699188 16324 raft_consensus.cc:2229] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:37.699294 16324 raft_consensus.cc:2258] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:37.702103 16324 ts_tablet_manager.cc:1875] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:37.703859 16324 ts_tablet_manager.cc:1888] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 01:59:37.703931 16324 log.cc:1196] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/wals/e6ac26fa6de44f7ca68e6c0b6b959db7
I20240502 01:59:37.704397 6629 catalog_manager.cc:4830] TS 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): tablet e6ac26fa6de44f7ca68e6c0b6b959db7 (table rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]) successfully deleted
I20240502 01:59:37.707484 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.825716 7050 consensus_queue.cc:237] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:37.827188 8642 raft_consensus.cc:1261] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:59:37.827730 16105 consensus_queue.cc:1035] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:37.837076 16105 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:37.840086 8642 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:37.840744 6639 catalog_manager.cc:5484] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 7 to 8, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:37.849472 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet 14e3c78c65334b52aca79d67c4bf45a2 with delete_type TABLET_DATA_TOMBSTONED (TS e7bcff2fef614e7a9981554c9e6db3c8 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59160
I20240502 01:59:37.851725 7050 consensus_queue.cc:237] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
W20240502 01:59:37.854528 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14e3c78c65334b52aca79d67c4bf45a2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:37.855877 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:37.860710 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:37.861872 8642 raft_consensus.cc:1261] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:59:37.862273 16105 consensus_queue.cc:1035] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:37.866012 16105 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:37.872582 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:37.873907 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.873919 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.875337 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 14e3c78c65334b52aca79d67c4bf45a2 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:59:37.876368 6639 catalog_manager.cc:5484] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 8 to 9, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:37.878132 8642 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:37.886667 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:37.892406 16431 tablet_replica.cc:330] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: stopping tablet replica
I20240502 01:59:37.892772 16431 raft_consensus.cc:2229] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:37.893046 16431 raft_consensus.cc:2258] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:37.893679 16431 ts_tablet_manager.cc:1875] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:37.896327 16431 ts_tablet_manager.cc:1888] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 01:59:37.901131 16431 log.cc:1196] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/14e3c78c65334b52aca79d67c4bf45a2
I20240502 01:59:37.901744 6630 catalog_manager.cc:4830] TS e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): tablet 14e3c78c65334b52aca79d67c4bf45a2 (table rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]) successfully deleted
I20240502 01:59:37.933866 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:37.944183 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.996085 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.998126 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:38.008467 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:38.010123 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.039220 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:38.042903 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.048908 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:38.052016 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:38.110904 16487 ts_tablet_manager.cc:902] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:38.116137 16487 tablet_copy_client.cc:308] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 01:59:38.116732 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet 44c55b199a24432e902cce9121fc0780 from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:47485)
I20240502 01:59:38.116868 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet 44c55b199a24432e902cce9121fc0780 from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:47485: session id = 954cd49326bf405abd81182d75dc783c-44c55b199a24432e902cce9121fc0780
I20240502 01:59:38.117965 6937 tablet_copy_source_session.cc:205] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:38.125532 16487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c55b199a24432e902cce9121fc0780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:38.129060 16487 tablet_copy_client.cc:783] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:38.129261 16487 tablet_copy_client.cc:647] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:38.138972 16487 tablet_copy_client.cc:523] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:38.148716 16487 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 01:59:38.151739 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.157517 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:38.159765 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:38.166574 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:38.166853 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.171381 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:38.171612 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.175877 16516 ts_tablet_manager.cc:902] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:38.179947 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:38.206909 16516 tablet_copy_client.cc:308] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:38.207592 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 91db5c1b3d2f4d22b5882a61ea0395c7 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 ({username='slave'} at 127.27.111.199:49969)
I20240502 01:59:38.207715 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 91db5c1b3d2f4d22b5882a61ea0395c7 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 at {username='slave'} at 127.27.111.199:49969: session id = 4e6e76b381af4cdc90bd3e9e6e8b5cf2-91db5c1b3d2f4d22b5882a61ea0395c7
I20240502 01:59:38.208817 7070 tablet_copy_source_session.cc:205] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:38.211336 16516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91db5c1b3d2f4d22b5882a61ea0395c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:38.214038 16516 tablet_copy_client.cc:783] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:38.214268 16516 tablet_copy_client.cc:647] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:38.218988 16487 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:38.219312 16487 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
I20240502 01:59:38.219638 16487 ts_tablet_manager.cc:1367] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.071s user 0.025s sys 0.003s
I20240502 01:59:38.220125 16487 raft_consensus.cc:348] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:38.220260 16487 raft_consensus.cc:729] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 01:59:38.220544 16487 consensus_queue.cc:260] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:38.220978 16487 ts_tablet_manager.cc:1398] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:38.221575 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-44c55b199a24432e902cce9121fc0780 received from {username='slave'} at 127.27.111.197:47485
I20240502 01:59:38.221661 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 954cd49326bf405abd81182d75dc783c-44c55b199a24432e902cce9121fc0780 on tablet 44c55b199a24432e902cce9121fc0780 with peer 954cd49326bf405abd81182d75dc783c
I20240502 01:59:38.253139 16516 tablet_copy_client.cc:523] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:38.254914 16516 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap starting.
I20240502 01:59:38.274159 16565 ts_tablet_manager.cc:902] T 14e3c78c65334b52aca79d67c4bf45a2 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:38.275455 16565 tablet_copy_client.cc:308] T 14e3c78c65334b52aca79d67c4bf45a2 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:38.276031 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 14e3c78c65334b52aca79d67c4bf45a2 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 ({username='slave'} at 127.27.111.199:49969)
I20240502 01:59:38.276146 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 14e3c78c65334b52aca79d67c4bf45a2 from peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 at {username='slave'} at 127.27.111.199:49969: session id = 4e6e76b381af4cdc90bd3e9e6e8b5cf2-14e3c78c65334b52aca79d67c4bf45a2
I20240502 01:59:38.277180 7070 tablet_copy_source_session.cc:205] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:38.279094 16565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e3c78c65334b52aca79d67c4bf45a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:38.284070 16565 tablet_copy_client.cc:783] T 14e3c78c65334b52aca79d67c4bf45a2 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:38.284237 16565 tablet_copy_client.cc:647] T 14e3c78c65334b52aca79d67c4bf45a2 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:38.285773 16565 tablet_copy_client.cc:523] T 14e3c78c65334b52aca79d67c4bf45a2 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:38.286939 16516 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:38.287268 16516 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap complete.
I20240502 01:59:38.287427 16565 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap starting.
I20240502 01:59:38.287477 16516 ts_tablet_manager.cc:1367] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.009s
I20240502 01:59:38.287976 16516 raft_consensus.cc:348] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:38.288105 16516 raft_consensus.cc:729] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, State: Initialized, Role: FOLLOWER
I20240502 01:59:38.288254 16516 consensus_queue.cc:260] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:38.288666 16516 ts_tablet_manager.cc:1398] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:38.289505 7048 consensus_queue.cc:237] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:38.290932 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-91db5c1b3d2f4d22b5882a61ea0395c7 received from {username='slave'} at 127.27.111.199:49969
I20240502 01:59:38.291013 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-91db5c1b3d2f4d22b5882a61ea0395c7 on tablet 91db5c1b3d2f4d22b5882a61ea0395c7 with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:38.291975 6916 raft_consensus.cc:1261] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:38.292454 8509 raft_consensus.cc:1261] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:38.293380 16105 consensus_queue.cc:1035] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:38.293685 16105 consensus_queue.cc:1035] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:59:38.294431 6984 consensus_peers.cc:485] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): Couldn't send request to peer 86bc8001c3464b929cd5c873861baa0c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:38.296679 7048 consensus_queue.cc:237] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:38.297921 8757 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:59:38.298154 8642 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 01:59:38.298350 6984 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): Couldn't send request to peer 86bc8001c3464b929cd5c873861baa0c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:38.298720 16110 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:38.301805 16110 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:38.302078 16110 raft_consensus.cc:2932] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.305366 6638 catalog_manager.cc:5484] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.306376 8509 raft_consensus.cc:2932] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.308422 6916 raft_consensus.cc:2932] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.319952 8757 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.321622 16599 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.324741 16565 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:38.325100 16565 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Bootstrap complete.
I20240502 01:59:38.325290 16565 ts_tablet_manager.cc:1367] T 14e3c78c65334b52aca79d67c4bf45a2 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent bootstrapping tablet: real 0.038s user 0.027s sys 0.000s
I20240502 01:59:38.325527 6639 catalog_manager.cc:5484] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 reported cstate change: config changed from index 7 to 8, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.325748 16565 raft_consensus.cc:348] T 14e3c78c65334b52aca79d67c4bf45a2 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:38.325841 16565 raft_consensus.cc:729] T 14e3c78c65334b52aca79d67c4bf45a2 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, State: Initialized, Role: FOLLOWER
I20240502 01:59:38.326001 16565 consensus_queue.cc:260] T 14e3c78c65334b52aca79d67c4bf45a2 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:38.326841 16565 ts_tablet_manager.cc:1398] T 14e3c78c65334b52aca79d67c4bf45a2 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 01:59:38.327085 8642 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.327339 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-14e3c78c65334b52aca79d67c4bf45a2 received from {username='slave'} at 127.27.111.199:49969
I20240502 01:59:38.327427 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 4e6e76b381af4cdc90bd3e9e6e8b5cf2-14e3c78c65334b52aca79d67c4bf45a2 on tablet 14e3c78c65334b52aca79d67c4bf45a2 with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:38.338335 6785 consensus_queue.cc:237] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:38.340251 8757 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240502 01:59:38.341686 6720 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e -> Peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): Couldn't send request to peer 86bc8001c3464b929cd5c873861baa0c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:38.342015 16297 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:38.344208 8509 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:59:38.344643 16297 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:38.346330 16297 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.349239 8757 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.350013 8509 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.352169 6643 catalog_manager.cc:5484] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 8 to 9, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New cstate: current_term: 2 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.383949 7048 consensus_queue.cc:237] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:38.389004 8760 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 01:59:38.389724 6984 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): Couldn't send request to peer 86bc8001c3464b929cd5c873861baa0c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:38.391112 8642 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 01:59:38.391709 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47a276e9780e448a9b3c6eab67925514. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:38.391937 16105 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:38.392350 16105 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:38.398463 16599 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:38.398900 8757 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:38.399202 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.402118 6639 catalog_manager.cc:5484] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 8 to 9, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:38.402602 8642 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:38.409030 6785 consensus_queue.cc:237] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:38.409996 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:38.410883 8508 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240502 01:59:38.412652 6720 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e -> Peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): Couldn't send request to peer 86bc8001c3464b929cd5c873861baa0c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:38.415172 16645 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:59:38.416085 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 01:59:38.423229 6721 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6ac26fa6de44f7ca68e6c0b6b959db7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:38.425050 8757 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:59:38.425998 16645 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:59:38.428189 16297 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:38.430716 8760 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:38.431087 8508 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:38.431138 6634 catalog_manager.cc:5484] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 9 to 10, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 2 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:38.433140 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.436897 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:38.458956 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:38.462152 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:38.472283 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.481706 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.491904 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:38.499066 8508 raft_consensus.cc:1203] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:59:38.507057 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.514827 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.516302 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:38.523450 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:38.537812 6785 consensus_queue.cc:237] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
W20240502 01:59:38.540760 6721 consensus_peers.cc:485] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f6c07e3923d47f29ef32561d7f438e1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:38.540833 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:38.541062 8903 raft_consensus.cc:1261] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:38.541541 6916 raft_consensus.cc:1261] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:38.541760 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:38.541924 16646 consensus_queue.cc:1035] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:38.542115 16645 consensus_queue.cc:1035] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:38.553946 6785 consensus_queue.cc:237] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:38.556573 8903 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:38.557080 16643 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:59:38.557248 6721 consensus_peers.cc:485] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0359dfbfd11746f1bf62a465f14634ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:38.557371 6917 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:38.558207 16643 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:38.565654 16643 raft_consensus.cc:2932] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:38.569391 8904 raft_consensus.cc:2932] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:38.569869 6639 catalog_manager.cc:5484] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:38.572464 6916 raft_consensus.cc:2932] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:38.585482 16646 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:38.587597 6917 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:38.590693 6634 catalog_manager.cc:5484] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 5 to 6, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:38.592334 8903 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:38.639607 8642 consensus_queue.cc:237] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:38.641047 7181 raft_consensus.cc:1261] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:38.641577 15840 consensus_queue.cc:1035] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:38.644439 15840 raft_consensus.cc:2932] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:38.646978 7181 raft_consensus.cc:2932] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:38.649377 6634 catalog_manager.cc:5484] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 5 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 2 leader_uuid: "4f53d80e39be410d92d4766692aff1c6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:38.652905 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet a66b932d884e4df8a78841103d88ded0 with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:38.653527 8642 consensus_queue.cc:237] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:38.656212 7181 raft_consensus.cc:1261] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:59:38.656594 16845 consensus_queue.cc:1035] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:59:38.657334 8577 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 -> Peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125): Couldn't send request to peer c6101231591a458384f4ea6d1f0e0ae5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a66b932d884e4df8a78841103d88ded0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:38.660221 8757 raft_consensus.cc:1203] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.8->[3.9-3.9] Dedup: 3.9->[]
I20240502 01:59:38.660696 15840 raft_consensus.cc:2932] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 01:59:38.663089 7048 consensus_queue.cc:237] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:38.663102 7181 raft_consensus.cc:2932] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 01:59:38.664034 8757 raft_consensus.cc:1203] T 14e3c78c65334b52aca79d67c4bf45a2 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 01:59:38.664899 6785 raft_consensus.cc:1261] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:38.665077 8508 raft_consensus.cc:1261] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:38.665519 16589 consensus_queue.cc:1035] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:38.665714 16682 consensus_queue.cc:1035] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:38.669281 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a66b932d884e4df8a78841103d88ded0 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:59:38.669775 6634 catalog_manager.cc:5484] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 7 to 8, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) added. New cstate: current_term: 2 leader_uuid: "4f53d80e39be410d92d4766692aff1c6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
W20240502 01:59:38.672590 6984 consensus_peers.cc:485] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): Couldn't send request to peer 86bc8001c3464b929cd5c873861baa0c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9313079c48c94ab1b7e022a91befafcd. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:38.691190 16847 tablet_replica.cc:330] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:38.692256 16105 raft_consensus.cc:2932] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.695848 6642 catalog_manager.cc:5484] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.696408 8508 raft_consensus.cc:2932] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.695299 6785 raft_consensus.cc:2932] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:38.704843 16847 raft_consensus.cc:2229] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:38.705067 16847 raft_consensus.cc:2258] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:38.708871 16847 ts_tablet_manager.cc:1875] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:38.713199 16847 ts_tablet_manager.cc:1888] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:59:38.713627 16847 log.cc:1196] T a66b932d884e4df8a78841103d88ded0 P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/a66b932d884e4df8a78841103d88ded0
I20240502 01:59:38.715107 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet a66b932d884e4df8a78841103d88ded0 (table rebalance_test_table_0 [id=6397b955acf741aeb8f869ab987d74b0]) successfully deleted
I20240502 01:59:38.727447 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.730705 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:38.735123 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.743260 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:38.753494 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:38.755477 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:38.757916 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:38.820163 16943 ts_tablet_manager.cc:902] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:38.821105 16943 tablet_copy_client.cc:308] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:38.821853 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet d4d429fe60e74108bc0c9ca8509365f2 from peer 86bc8001c3464b929cd5c873861baa0c ({username='slave'} at 127.27.111.196:45961)
I20240502 01:59:38.821970 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet d4d429fe60e74108bc0c9ca8509365f2 from peer 86bc8001c3464b929cd5c873861baa0c at {username='slave'} at 127.27.111.196:45961: session id = 86bc8001c3464b929cd5c873861baa0c-d4d429fe60e74108bc0c9ca8509365f2
I20240502 01:59:38.823192 7070 tablet_copy_source_session.cc:205] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:38.823566 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.827226 16943 ts_tablet_manager.cc:1875] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:59:38.837904 16943 ts_tablet_manager.cc:1888] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 01:59:38.838276 16943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d429fe60e74108bc0c9ca8509365f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:38.840497 16943 tablet_copy_client.cc:783] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:38.840775 16943 tablet_copy_client.cc:647] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:38.842362 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.843324 16943 tablet_copy_client.cc:523] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:38.845167 16943 tablet_bootstrap.cc:492] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:38.860137 16970 ts_tablet_manager.cc:902] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:38.861641 16970 tablet_copy_client.cc:308] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:38.862816 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 47a276e9780e448a9b3c6eab67925514 from peer 86bc8001c3464b929cd5c873861baa0c ({username='slave'} at 127.27.111.196:45961)
I20240502 01:59:38.862938 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 47a276e9780e448a9b3c6eab67925514 from peer 86bc8001c3464b929cd5c873861baa0c at {username='slave'} at 127.27.111.196:45961: session id = 86bc8001c3464b929cd5c873861baa0c-47a276e9780e448a9b3c6eab67925514
I20240502 01:59:38.864143 7070 tablet_copy_source_session.cc:205] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:38.865347 16970 ts_tablet_manager.cc:1875] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:59:38.867635 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.868301 16970 ts_tablet_manager.cc:1888] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 01:59:38.868563 16970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a276e9780e448a9b3c6eab67925514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:38.871187 16970 tablet_copy_client.cc:783] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:38.871384 16970 tablet_copy_client.cc:647] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:38.873411 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:38.879868 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.882933 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:38.885105 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:38.886492 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:38.889005 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.907518 16970 tablet_copy_client.cc:523] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:38.909195 16970 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:38.918190 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.929790 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:38.930809 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.939875 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:38.945834 16970 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:38.946200 16970 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap complete.
I20240502 01:59:38.946435 16970 ts_tablet_manager.cc:1367] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.037s user 0.027s sys 0.000s
I20240502 01:59:38.947461 16970 raft_consensus.cc:348] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:38.947620 16970 raft_consensus.cc:729] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:38.947831 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:38.947803 16970 consensus_queue.cc:260] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:38.948204 16970 ts_tablet_manager.cc:1398] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:38.948740 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 86bc8001c3464b929cd5c873861baa0c-47a276e9780e448a9b3c6eab67925514 received from {username='slave'} at 127.27.111.196:45961
I20240502 01:59:38.948840 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 86bc8001c3464b929cd5c873861baa0c-47a276e9780e448a9b3c6eab67925514 on tablet 47a276e9780e448a9b3c6eab67925514 with peer 86bc8001c3464b929cd5c873861baa0c
I20240502 01:59:38.958272 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:38.965548 16970 ts_tablet_manager.cc:902] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:38.966141 16970 tablet_copy_client.cc:308] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:38.966735 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet e6ac26fa6de44f7ca68e6c0b6b959db7 from peer 86bc8001c3464b929cd5c873861baa0c ({username='slave'} at 127.27.111.196:45315)
I20240502 01:59:38.966852 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet e6ac26fa6de44f7ca68e6c0b6b959db7 from peer 86bc8001c3464b929cd5c873861baa0c at {username='slave'} at 127.27.111.196:45315: session id = 86bc8001c3464b929cd5c873861baa0c-e6ac26fa6de44f7ca68e6c0b6b959db7
I20240502 01:59:38.967793 6805 tablet_copy_source_session.cc:205] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:38.968684 16970 ts_tablet_manager.cc:1875] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:59:38.971567 16970 ts_tablet_manager.cc:1888] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.7
I20240502 01:59:38.971800 16970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ac26fa6de44f7ca68e6c0b6b959db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:38.973204 16970 tablet_copy_client.cc:783] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:38.973412 16970 tablet_copy_client.cc:647] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:38.975414 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:38.976635 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:38.991735 17004 ts_tablet_manager.cc:902] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:38.992472 17004 tablet_copy_client.cc:308] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:38.994506 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet 0359dfbfd11746f1bf62a465f14634ec from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:33267)
I20240502 01:59:38.994629 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet 0359dfbfd11746f1bf62a465f14634ec from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:33267: session id = 954cd49326bf405abd81182d75dc783c-0359dfbfd11746f1bf62a465f14634ec
I20240502 01:59:38.995607 6805 tablet_copy_source_session.cc:205] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:38.996603 17004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0359dfbfd11746f1bf62a465f14634ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:39.001858 16970 tablet_copy_client.cc:523] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:39.003746 16970 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:39.004993 17004 tablet_copy_client.cc:783] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:39.005246 17004 tablet_copy_client.cc:647] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:39.012099 17010 ts_tablet_manager.cc:902] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:39.012835 17010 tablet_copy_client.cc:308] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:39.013190 17011 ts_tablet_manager.cc:902] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:39.013904 17011 tablet_copy_client.cc:308] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:39.014433 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 47a276e9780e448a9b3c6eab67925514 from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:40005)
I20240502 01:59:39.014459 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet e6ac26fa6de44f7ca68e6c0b6b959db7 from peer 4f53d80e39be410d92d4766692aff1c6 ({username='slave'} at 127.27.111.198:60983)
I20240502 01:59:39.014542 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 47a276e9780e448a9b3c6eab67925514 from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:40005: session id = 954cd49326bf405abd81182d75dc783c-47a276e9780e448a9b3c6eab67925514
I20240502 01:59:39.014564 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet e6ac26fa6de44f7ca68e6c0b6b959db7 from peer 4f53d80e39be410d92d4766692aff1c6 at {username='slave'} at 127.27.111.198:60983: session id = 4f53d80e39be410d92d4766692aff1c6-e6ac26fa6de44f7ca68e6c0b6b959db7
I20240502 01:59:39.015470 6805 tablet_copy_source_session.cc:205] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:39.015504 7070 tablet_copy_source_session.cc:205] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:39.016490 17011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6ac26fa6de44f7ca68e6c0b6b959db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:39.020946 17004 tablet_copy_client.cc:523] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:39.022553 17004 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 01:59:39.022926 17010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a276e9780e448a9b3c6eab67925514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:39.026477 17010 tablet_copy_client.cc:783] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:39.026685 17010 tablet_copy_client.cc:647] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:39.028635 17011 tablet_copy_client.cc:783] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:39.028801 17011 tablet_copy_client.cc:647] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:39.036846 16943 tablet_bootstrap.cc:492] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:39.037268 16943 tablet_bootstrap.cc:492] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap complete.
I20240502 01:59:39.037528 16943 ts_tablet_manager.cc:1367] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.192s user 0.031s sys 0.000s
I20240502 01:59:39.038005 16943 raft_consensus.cc:348] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:39.038224 16943 raft_consensus.cc:729] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:39.038450 16943 consensus_queue.cc:260] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:39.039647 16943 ts_tablet_manager.cc:1398] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:39.042982 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 86bc8001c3464b929cd5c873861baa0c-d4d429fe60e74108bc0c9ca8509365f2 received from {username='slave'} at 127.27.111.196:45961
I20240502 01:59:39.043077 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 86bc8001c3464b929cd5c873861baa0c-d4d429fe60e74108bc0c9ca8509365f2 on tablet d4d429fe60e74108bc0c9ca8509365f2 with peer 86bc8001c3464b929cd5c873861baa0c
I20240502 01:59:39.067744 17004 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:39.068094 17004 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
I20240502 01:59:39.068284 17004 ts_tablet_manager.cc:1367] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.046s user 0.024s sys 0.004s
I20240502 01:59:39.068734 17004 raft_consensus.cc:348] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:39.068872 17004 raft_consensus.cc:729] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 01:59:39.069036 17004 consensus_queue.cc:260] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:39.069425 17004 ts_tablet_manager.cc:1398] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:59:39.069962 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-0359dfbfd11746f1bf62a465f14634ec received from {username='slave'} at 127.27.111.197:33267
I20240502 01:59:39.070042 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 954cd49326bf405abd81182d75dc783c-0359dfbfd11746f1bf62a465f14634ec on tablet 0359dfbfd11746f1bf62a465f14634ec with peer 954cd49326bf405abd81182d75dc783c
I20240502 01:59:39.079252 17010 tablet_copy_client.cc:523] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:39.079337 17011 tablet_copy_client.cc:523] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:39.080751 17010 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 01:59:39.080751 17011 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap starting.
I20240502 01:59:39.100952 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.114264 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.119498 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:39.120476 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:39.129678 17011 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:39.130044 17011 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap complete.
I20240502 01:59:39.130244 17011 ts_tablet_manager.cc:1367] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6: Time spent bootstrapping tablet: real 0.050s user 0.028s sys 0.005s
I20240502 01:59:39.130245 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:39.132064 17011 raft_consensus.cc:348] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:39.132205 17011 raft_consensus.cc:729] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Initialized, Role: FOLLOWER
I20240502 01:59:39.132342 17011 consensus_queue.cc:260] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:39.132624 17011 ts_tablet_manager.cc:1398] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:59:39.133070 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 4f53d80e39be410d92d4766692aff1c6-e6ac26fa6de44f7ca68e6c0b6b959db7 received from {username='slave'} at 127.27.111.198:60983
I20240502 01:59:39.133153 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 4f53d80e39be410d92d4766692aff1c6-e6ac26fa6de44f7ca68e6c0b6b959db7 on tablet e6ac26fa6de44f7ca68e6c0b6b959db7 with peer 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:39.139286 16943 ts_tablet_manager.cc:902] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:39.139744 16943 tablet_copy_client.cc:308] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:39.140143 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 9313079c48c94ab1b7e022a91befafcd from peer 86bc8001c3464b929cd5c873861baa0c ({username='slave'} at 127.27.111.196:45961)
I20240502 01:59:39.140239 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 9313079c48c94ab1b7e022a91befafcd from peer 86bc8001c3464b929cd5c873861baa0c at {username='slave'} at 127.27.111.196:45961: session id = 86bc8001c3464b929cd5c873861baa0c-9313079c48c94ab1b7e022a91befafcd
I20240502 01:59:39.141158 7070 tablet_copy_source_session.cc:205] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:39.142223 16943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9313079c48c94ab1b7e022a91befafcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:39.142172 6917 consensus_queue.cc:237] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:39.144682 16943 tablet_copy_client.cc:783] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:39.144834 16943 tablet_copy_client.cc:647] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:39.144829 8508 raft_consensus.cc:1261] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 01:59:39.145177 6851 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): Couldn't send request to peer 85e78945910a4e1abfa101d2edc3644e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:39.145545 8903 raft_consensus.cc:1261] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:59:39.145943 16232 consensus_queue.cc:1035] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:39.146253 16232 consensus_queue.cc:1035] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:39.148145 17010 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:39.148434 17010 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
I20240502 01:59:39.148619 17010 ts_tablet_manager.cc:1367] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.068s user 0.029s sys 0.000s
I20240502 01:59:39.149138 17010 raft_consensus.cc:348] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:39.149263 17010 raft_consensus.cc:729] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 01:59:39.149281 16970 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:39.149436 17010 consensus_queue.cc:260] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:39.149531 16970 tablet_bootstrap.cc:492] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Bootstrap complete.
I20240502 01:59:39.149690 16970 ts_tablet_manager.cc:1367] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.146s user 0.027s sys 0.000s
I20240502 01:59:39.149837 17010 ts_tablet_manager.cc:1398] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 01:59:39.150100 16970 raft_consensus.cc:348] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:39.150235 16970 raft_consensus.cc:729] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:39.150349 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-47a276e9780e448a9b3c6eab67925514 received from {username='slave'} at 127.27.111.197:40005
I20240502 01:59:39.150451 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 954cd49326bf405abd81182d75dc783c-47a276e9780e448a9b3c6eab67925514 on tablet 47a276e9780e448a9b3c6eab67925514 with peer 954cd49326bf405abd81182d75dc783c
I20240502 01:59:39.150373 16970 consensus_queue.cc:260] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:39.151168 17105 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:39.151616 8903 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:39.154811 8508 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:39.157343 6637 catalog_manager.cc:5484] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 7 to 8, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:39.163875 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.164090 16943 tablet_copy_client.cc:523] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:39.150715 16970 ts_tablet_manager.cc:1398] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240502 01:59:39.165805 16943 tablet_bootstrap.cc:492] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c: Bootstrap starting.
I20240502 01:59:39.165839 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 86bc8001c3464b929cd5c873861baa0c-e6ac26fa6de44f7ca68e6c0b6b959db7 received from {username='slave'} at 127.27.111.196:45315
I20240502 01:59:39.165966 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 86bc8001c3464b929cd5c873861baa0c-e6ac26fa6de44f7ca68e6c0b6b959db7 on tablet e6ac26fa6de44f7ca68e6c0b6b959db7 with peer 86bc8001c3464b929cd5c873861baa0c
I20240502 01:59:39.168965 17011 ts_tablet_manager.cc:902] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:39.169492 17011 tablet_copy_client.cc:308] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:39.170090 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet 9f6c07e3923d47f29ef32561d7f438e1 from peer 4f53d80e39be410d92d4766692aff1c6 ({username='slave'} at 127.27.111.198:60983)
I20240502 01:59:39.170202 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet 9f6c07e3923d47f29ef32561d7f438e1 from peer 4f53d80e39be410d92d4766692aff1c6 at {username='slave'} at 127.27.111.198:60983: session id = 4f53d80e39be410d92d4766692aff1c6-9f6c07e3923d47f29ef32561d7f438e1
I20240502 01:59:39.171260 6805 tablet_copy_source_session.cc:205] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:39.179325 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:39.188292 17011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f6c07e3923d47f29ef32561d7f438e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:39.191009 17011 tablet_copy_client.cc:783] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:39.191262 17011 tablet_copy_client.cc:647] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:39.192715 17011 tablet_copy_client.cc:523] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:39.194284 17011 tablet_bootstrap.cc:492] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap starting.
I20240502 01:59:39.222362 17011 tablet_bootstrap.cc:492] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:39.222810 17011 tablet_bootstrap.cc:492] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap complete.
I20240502 01:59:39.223023 17011 ts_tablet_manager.cc:1367] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.001s
I20240502 01:59:39.223497 17011 raft_consensus.cc:348] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:39.223626 17011 raft_consensus.cc:729] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Initialized, Role: FOLLOWER
I20240502 01:59:39.223788 17011 consensus_queue.cc:260] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:39.224064 17011 ts_tablet_manager.cc:1398] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 01:59:39.227030 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 4f53d80e39be410d92d4766692aff1c6-9f6c07e3923d47f29ef32561d7f438e1 received from {username='slave'} at 127.27.111.198:60983
I20240502 01:59:39.227138 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 4f53d80e39be410d92d4766692aff1c6-9f6c07e3923d47f29ef32561d7f438e1 on tablet 9f6c07e3923d47f29ef32561d7f438e1 with peer 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:39.235179 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:39.257237 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.263306 16943 tablet_bootstrap.cc:492] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:39.263710 16943 tablet_bootstrap.cc:492] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c: Bootstrap complete.
I20240502 01:59:39.263964 16943 ts_tablet_manager.cc:1367] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c: Time spent bootstrapping tablet: real 0.098s user 0.027s sys 0.003s
I20240502 01:59:39.264498 16943 raft_consensus.cc:348] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:39.264596 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:39.264657 16943 raft_consensus.cc:729] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Initialized, Role: FOLLOWER
I20240502 01:59:39.264799 16943 consensus_queue.cc:260] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:39.265125 16943 ts_tablet_manager.cc:1398] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:39.265664 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 86bc8001c3464b929cd5c873861baa0c-9313079c48c94ab1b7e022a91befafcd received from {username='slave'} at 127.27.111.196:45961
I20240502 01:59:39.265750 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 86bc8001c3464b929cd5c873861baa0c-9313079c48c94ab1b7e022a91befafcd on tablet 9313079c48c94ab1b7e022a91befafcd with peer 86bc8001c3464b929cd5c873861baa0c
I20240502 01:59:39.265851 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.273294 7181 consensus_queue.cc:237] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:39.274879 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:39.275322 6917 raft_consensus.cc:1261] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:39.275607 8903 raft_consensus.cc:1261] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:59:39.275837 15568 consensus_queue.cc:1035] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:59:39.276073 7116 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): Couldn't send request to peer 85e78945910a4e1abfa101d2edc3644e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7fa358140eb742b587a4d6a203738b4f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:39.276249 15568 consensus_queue.cc:1035] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:39.310149 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.311700 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:39.315351 17159 ts_tablet_manager.cc:902] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5: Initiating tablet copy from peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
I20240502 01:59:39.316104 17159 tablet_copy_client.cc:308] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.198:46475
I20240502 01:59:39.318138 8662 tablet_copy_service.cc:140] P 4f53d80e39be410d92d4766692aff1c6: Received BeginTabletCopySession request for tablet a66b932d884e4df8a78841103d88ded0 from peer c6101231591a458384f4ea6d1f0e0ae5 ({username='slave'} at 127.27.111.195:38535)
I20240502 01:59:39.318244 8662 tablet_copy_service.cc:161] P 4f53d80e39be410d92d4766692aff1c6: Beginning new tablet copy session on tablet a66b932d884e4df8a78841103d88ded0 from peer c6101231591a458384f4ea6d1f0e0ae5 at {username='slave'} at 127.27.111.195:38535: session id = c6101231591a458384f4ea6d1f0e0ae5-a66b932d884e4df8a78841103d88ded0
I20240502 01:59:39.319103 8662 tablet_copy_source_session.cc:205] T a66b932d884e4df8a78841103d88ded0 P 4f53d80e39be410d92d4766692aff1c6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:39.322118 17162 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:39.322352 7181 raft_consensus.cc:1203] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 01:59:39.323791 6917 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:39.325210 8903 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:39.325615 17159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a66b932d884e4df8a78841103d88ded0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:39.327937 6640 catalog_manager.cc:5484] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 5 to 6, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:39.332309 17159 tablet_copy_client.cc:783] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:39.332576 17159 tablet_copy_client.cc:647] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:39.333261 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:39.346307 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:39.375353 17159 tablet_copy_client.cc:523] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:39.378139 17159 tablet_bootstrap.cc:492] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 01:59:39.390300 7181 raft_consensus.cc:1203] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:59:39.407750 6785 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a0548274c47148d9b611fd3c978a0c34"
dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49020
I20240502 01:59:39.407908 6785 raft_consensus.cc:595] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Received request to transfer leadership
I20240502 01:59:39.428937 17159 tablet_bootstrap.cc:492] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:39.430907 17159 tablet_bootstrap.cc:492] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap complete.
I20240502 01:59:39.431136 17159 ts_tablet_manager.cc:1367] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.053s user 0.030s sys 0.000s
I20240502 01:59:39.438423 8642 raft_consensus.cc:1203] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240502 01:59:39.432040 17159 raft_consensus.cc:348] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:39.445106 17159 raft_consensus.cc:729] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 01:59:39.447854 17159 consensus_queue.cc:260] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 01:59:39.448374 17159 ts_tablet_manager.cc:1398] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20240502 01:59:39.449258 8662 tablet_copy_service.cc:342] P 4f53d80e39be410d92d4766692aff1c6: Request end of tablet copy session c6101231591a458384f4ea6d1f0e0ae5-a66b932d884e4df8a78841103d88ded0 received from {username='slave'} at 127.27.111.195:38535
I20240502 01:59:39.449355 8662 tablet_copy_service.cc:433] P 4f53d80e39be410d92d4766692aff1c6: ending tablet copy session c6101231591a458384f4ea6d1f0e0ae5-a66b932d884e4df8a78841103d88ded0 on tablet a66b932d884e4df8a78841103d88ded0 with peer c6101231591a458384f4ea6d1f0e0ae5
I20240502 01:59:39.453987 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.458664 6763 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:39.464826 8508 raft_consensus.cc:1203] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 01:59:39.472018 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.478268 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:39.482085 7181 raft_consensus.cc:1203] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240502 01:59:39.483954 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.484738 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:39.499030 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.500638 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:39.503266 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:39.515879 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.520335 8508 raft_consensus.cc:1203] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:59:39.521446 7181 raft_consensus.cc:1203] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:59:39.532712 7030 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:59:39.537094 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:39.563421 8642 raft_consensus.cc:1203] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:59:39.571102 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:39.577006 16643 raft_consensus.cc:979] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: : Instructing follower c6101231591a458384f4ea6d1f0e0ae5 to start an election
I20240502 01:59:39.577101 16643 raft_consensus.cc:1067] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Signalling peer c6101231591a458384f4ea6d1f0e0ae5 to start an election
I20240502 01:59:39.578308 7048 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a0548274c47148d9b611fd3c978a0c34"
dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
from {username='slave'} at 127.27.111.193:42103
I20240502 01:59:39.578482 7048 raft_consensus.cc:482] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:39.578541 7048 raft_consensus.cc:3037] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:39.579735 7048 raft_consensus.cc:504] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:39.580176 7048 leader_election.cc:288] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 2 election: Requested vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741)
I20240502 01:59:39.580704 8757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0548274c47148d9b611fd3c978a0c34" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2"
I20240502 01:59:39.580848 8757 raft_consensus.cc:3037] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:39.582001 8757 raft_consensus.cc:2454] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 2.
I20240502 01:59:39.582742 6982 leader_election.cc:302] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e6e76b381af4cdc90bd3e9e6e8b5cf2, c6101231591a458384f4ea6d1f0e0ae5; no voters:
I20240502 01:59:39.583681 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0548274c47148d9b611fd3c978a0c34" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:39.583815 6785 raft_consensus.cc:3032] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:39.583858 6785 raft_consensus.cc:729] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:39.584000 6785 consensus_queue.cc:260] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:39.584143 6785 raft_consensus.cc:3037] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:39.585189 6785 raft_consensus.cc:2454] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6101231591a458384f4ea6d1f0e0ae5 in term 2.
I20240502 01:59:39.589349 6763 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:39.592787 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:39.596303 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.608180 17309 ts_tablet_manager.cc:902] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 01:59:39.611189 17289 raft_consensus.cc:2781] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:39.615120 17289 raft_consensus.cc:686] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 LEADER]: Becoming Leader. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Running, Role: LEADER
I20240502 01:59:39.615643 17289 consensus_queue.cc:237] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:39.618279 17309 tablet_copy_client.cc:308] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 01:59:39.619503 6642 catalog_manager.cc:5484] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: term changed from 1 to 2, leader changed from 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) to c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195). New cstate: current_term: 2 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:39.622068 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 from peer 85e78945910a4e1abfa101d2edc3644e ({username='slave'} at 127.27.111.193:40059)
I20240502 01:59:39.622191 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 from peer 85e78945910a4e1abfa101d2edc3644e at {username='slave'} at 127.27.111.193:40059: session id = 85e78945910a4e1abfa101d2edc3644e-9ecbf1c8ee8c46b195d2374a5db81ac0
I20240502 01:59:39.623132 6937 tablet_copy_source_session.cc:205] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:39.623924 17309 ts_tablet_manager.cc:1875] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:59:39.626374 17309 ts_tablet_manager.cc:1888] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 01:59:39.626592 17309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:39.629004 17309 tablet_copy_client.cc:783] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:39.629235 17309 tablet_copy_client.cc:647] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:39.635115 17309 tablet_copy_client.cc:523] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:39.636813 17309 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:39.675352 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.676389 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:39.679061 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:39.679783 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.682574 6763 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:39.688611 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:39.700616 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.716524 6894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:39.742352 6917 consensus_queue.cc:237] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:39.744696 8760 raft_consensus.cc:1261] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:39.745901 17105 consensus_queue.cc:1035] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:39.746963 8903 raft_consensus.cc:1261] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:39.747540 17105 consensus_queue.cc:1035] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:39.750057 17105 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:39.752578 8903 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:39.754356 6642 catalog_manager.cc:5484] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 6 to 7, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:39.767632 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc5daef41bc44c4855f2bcb44e1c6d5 with delete_type TABLET_DATA_TOMBSTONED (TS c6101231591a458384f4ea6d1f0e0ae5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36030
I20240502 01:59:39.779888 17379 ts_tablet_manager.cc:902] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 01:59:39.784776 17382 tablet_replica.cc:330] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5: stopping tablet replica
I20240502 01:59:39.785442 17382 raft_consensus.cc:2229] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:39.786365 17382 raft_consensus.cc:2258] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:39.787127 17379 tablet_copy_client.cc:308] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:36569
I20240502 01:59:39.787746 7201 tablet_copy_service.cc:140] P 86bc8001c3464b929cd5c873861baa0c: Received BeginTabletCopySession request for tablet 7fa358140eb742b587a4d6a203738b4f from peer 85e78945910a4e1abfa101d2edc3644e ({username='slave'} at 127.27.111.193:43151)
I20240502 01:59:39.787865 7201 tablet_copy_service.cc:161] P 86bc8001c3464b929cd5c873861baa0c: Beginning new tablet copy session on tablet 7fa358140eb742b587a4d6a203738b4f from peer 85e78945910a4e1abfa101d2edc3644e at {username='slave'} at 127.27.111.193:43151: session id = 85e78945910a4e1abfa101d2edc3644e-7fa358140eb742b587a4d6a203738b4f
I20240502 01:59:39.788700 17382 ts_tablet_manager.cc:1875] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:39.788851 7201 tablet_copy_source_session.cc:205] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:39.789325 8760 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:39.791353 17382 ts_tablet_manager.cc:1888] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:59:39.791484 17382 log.cc:1196] T dcc5daef41bc44c4855f2bcb44e1c6d5 P c6101231591a458384f4ea6d1f0e0ae5: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/dcc5daef41bc44c4855f2bcb44e1c6d5
I20240502 01:59:39.791496 17309 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:39.792179 6630 catalog_manager.cc:4830] TS c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125): tablet dcc5daef41bc44c4855f2bcb44e1c6d5 (table rebalance_test_table_0 [id=6397b955acf741aeb8f869ab987d74b0]) successfully deleted
I20240502 01:59:39.792361 17379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fa358140eb742b587a4d6a203738b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:39.795586 6917 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "17235e019e644b258266636fb6be6648"
dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46184
I20240502 01:59:39.795758 6917 raft_consensus.cc:595] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:59:39.798537 17309 tablet_bootstrap.cc:492] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap complete.
I20240502 01:59:39.798902 17309 ts_tablet_manager.cc:1367] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.162s user 0.026s sys 0.005s
I20240502 01:59:39.799346 17309 raft_consensus.cc:348] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:39.799459 17309 raft_consensus.cc:729] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:39.799594 17309 consensus_queue.cc:260] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:39.799887 17309 ts_tablet_manager.cc:1398] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:59:39.808893 7048 raft_consensus.cc:1203] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 01:59:39.811424 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 85e78945910a4e1abfa101d2edc3644e-9ecbf1c8ee8c46b195d2374a5db81ac0 received from {username='slave'} at 127.27.111.193:40059
I20240502 01:59:39.811517 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 85e78945910a4e1abfa101d2edc3644e-9ecbf1c8ee8c46b195d2374a5db81ac0 on tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 with peer 85e78945910a4e1abfa101d2edc3644e
I20240502 01:59:39.815894 17379 tablet_copy_client.cc:783] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:39.816217 17379 tablet_copy_client.cc:647] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:39.817487 17379 tablet_copy_client.cc:523] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:39.819051 17379 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:39.837373 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:39.839766 6763 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:39.839916 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.840442 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:39.840891 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.848917 8489 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.887671 17106 raft_consensus.cc:979] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8: : Instructing follower 954cd49326bf405abd81182d75dc783c to start an election
I20240502 01:59:39.887784 17106 raft_consensus.cc:1067] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Signalling peer 954cd49326bf405abd81182d75dc783c to start an election
I20240502 01:59:39.888537 8508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17235e019e644b258266636fb6be6648"
dest_uuid: "954cd49326bf405abd81182d75dc783c"
from {username='slave'} at 127.27.111.194:55631
I20240502 01:59:39.888759 8508 raft_consensus.cc:482] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:39.888854 8508 raft_consensus.cc:3037] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:39.888971 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.890082 8508 raft_consensus.cc:504] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:39.891384 8508 leader_election.cc:288] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 2 election: Requested vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
I20240502 01:59:39.892861 7048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17235e019e644b258266636fb6be6648" candidate_uuid: "954cd49326bf405abd81182d75dc783c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 01:59:39.892992 7048 raft_consensus.cc:3037] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:39.893666 6763 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:39.894155 7048 raft_consensus.cc:2454] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 954cd49326bf405abd81182d75dc783c in term 2.
I20240502 01:59:39.894580 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17235e019e644b258266636fb6be6648" candidate_uuid: "954cd49326bf405abd81182d75dc783c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:39.894707 6917 raft_consensus.cc:3032] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:39.894752 6917 raft_consensus.cc:729] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Running, Role: LEADER
I20240502 01:59:39.894942 6917 consensus_queue.cc:260] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:39.895167 6917 raft_consensus.cc:3037] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:39.896260 6917 raft_consensus.cc:2454] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 954cd49326bf405abd81182d75dc783c in term 2.
I20240502 01:59:39.897200 8445 leader_election.cc:302] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 954cd49326bf405abd81182d75dc783c, c6101231591a458384f4ea6d1f0e0ae5, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:39.897452 16541 raft_consensus.cc:2781] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:39.897599 16541 raft_consensus.cc:686] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [term 2 LEADER]: Becoming Leader. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Running, Role: LEADER
I20240502 01:59:39.897934 16541 consensus_queue.cc:237] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:39.901309 6635 catalog_manager.cc:5484] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c reported cstate change: term changed from 1 to 2, leader changed from e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) to 954cd49326bf405abd81182d75dc783c (127.27.111.197). New cstate: current_term: 2 leader_uuid: "954cd49326bf405abd81182d75dc783c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:39.908828 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.915601 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:39.919234 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:39.919598 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:39.920941 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:39.941594 8641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17235e019e644b258266636fb6be6648" candidate_uuid: "954cd49326bf405abd81182d75dc783c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "4f53d80e39be410d92d4766692aff1c6"
I20240502 01:59:39.941749 8641 raft_consensus.cc:3037] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:39.941871 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:39.943787 8641 raft_consensus.cc:2454] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 954cd49326bf405abd81182d75dc783c in term 2.
I20240502 01:59:39.952965 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:39.961911 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:39.999624 6785 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:59:40.000294 17289 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:40.005744 8760 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:59:40.006719 16110 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:59:40.010159 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.025136 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.025568 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:40.039186 17379 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:40.039606 17379 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e: Bootstrap complete.
I20240502 01:59:40.039856 17379 ts_tablet_manager.cc:1367] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.221s user 0.024s sys 0.004s
I20240502 01:59:40.040385 17379 raft_consensus.cc:348] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:40.040608 17379 raft_consensus.cc:729] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:40.040772 17379 consensus_queue.cc:260] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:40.041121 17379 ts_tablet_manager.cc:1398] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:40.041694 7201 tablet_copy_service.cc:342] P 86bc8001c3464b929cd5c873861baa0c: Request end of tablet copy session 85e78945910a4e1abfa101d2edc3644e-7fa358140eb742b587a4d6a203738b4f received from {username='slave'} at 127.27.111.193:43151
I20240502 01:59:40.041790 7201 tablet_copy_service.cc:433] P 86bc8001c3464b929cd5c873861baa0c: ending tablet copy session 85e78945910a4e1abfa101d2edc3644e-7fa358140eb742b587a4d6a203738b4f on tablet 7fa358140eb742b587a4d6a203738b4f with peer 85e78945910a4e1abfa101d2edc3644e
I20240502 01:59:40.064705 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:40.065670 6763 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.077100 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.103109 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.103601 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:40.108698 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.116128 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:40.119442 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:40.133476 6763 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.139917 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.149547 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:40.160537 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:40.161022 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:40.228130 6785 raft_consensus.cc:1203] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:59:40.243583 6785 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7"
dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49102
I20240502 01:59:40.243744 6785 raft_consensus.cc:595] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Received request to transfer leadership
I20240502 01:59:40.248757 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.250384 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:40.261687 6763 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.262959 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:40.264418 6783 raft_consensus.cc:1203] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 01:59:40.274508 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.276641 7161 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:40.277729 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:40.285483 6917 raft_consensus.cc:1261] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Refusing update from remote peer 954cd49326bf405abd81182d75dc783c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:40.286001 17464 consensus_queue.cc:1035] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:40.292224 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.325242 17292 raft_consensus.cc:979] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e: : Instructing follower 86bc8001c3464b929cd5c873861baa0c to start an election
I20240502 01:59:40.325347 17292 raft_consensus.cc:1067] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Signalling peer 86bc8001c3464b929cd5c873861baa0c to start an election
I20240502 01:59:40.325875 7181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7"
dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
from {username='slave'} at 127.27.111.193:43151
I20240502 01:59:40.326009 7181 raft_consensus.cc:482] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:40.326058 7181 raft_consensus.cc:3037] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:40.327198 7181 raft_consensus.cc:504] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:40.327746 7181 leader_election.cc:288] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 election: Requested vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741), 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
I20240502 01:59:40.327970 6783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:40.328099 6783 raft_consensus.cc:3032] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:59:40.328140 6783 raft_consensus.cc:729] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:40.328291 6783 consensus_queue.cc:260] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:40.328541 6783 raft_consensus.cc:3037] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:40.328765 8508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "954cd49326bf405abd81182d75dc783c"
I20240502 01:59:40.328893 8508 raft_consensus.cc:3037] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:40.329000 8641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "4f53d80e39be410d92d4766692aff1c6"
I20240502 01:59:40.329120 8641 raft_consensus.cc:3037] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:40.329612 6783 raft_consensus.cc:2454] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 3.
I20240502 01:59:40.330004 8508 raft_consensus.cc:2454] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 3.
I20240502 01:59:40.330164 8641 raft_consensus.cc:2454] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 3.
I20240502 01:59:40.330785 7117 leader_election.cc:302] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 4f53d80e39be410d92d4766692aff1c6, 86bc8001c3464b929cd5c873861baa0c, 954cd49326bf405abd81182d75dc783c; no voters:
I20240502 01:59:40.331131 17184 raft_consensus.cc:2781] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:59:40.331311 17184 raft_consensus.cc:686] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 3 LEADER]: Becoming Leader. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Running, Role: LEADER
I20240502 01:59:40.331480 17184 consensus_queue.cc:237] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:40.333880 8760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6ac26fa6de44f7ca68e6c0b6b959db7" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2"
I20240502 01:59:40.334013 8760 raft_consensus.cc:3037] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:40.333879 6642 catalog_manager.cc:5484] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: term changed from 2 to 3, leader changed from 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) to 86bc8001c3464b929cd5c873861baa0c (127.27.111.196). New cstate: current_term: 3 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:40.335130 8760 raft_consensus.cc:2454] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 3.
I20240502 01:59:40.375173 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.375476 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.378320 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:40.381659 6763 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:40.386186 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.401862 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:40.402148 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.402721 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:40.405282 7048 raft_consensus.cc:1261] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Refusing update from remote peer 954cd49326bf405abd81182d75dc783c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:40.405838 17464 consensus_queue.cc:1035] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:40.432744 8641 raft_consensus.cc:1261] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Refusing update from remote peer 954cd49326bf405abd81182d75dc783c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:40.437376 17464 consensus_queue.cc:1035] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:40.457232 8641 consensus_queue.cc:237] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:40.458840 8903 raft_consensus.cc:1261] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:59:40.459432 17376 consensus_queue.cc:1035] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:59:40.460686 7182 raft_consensus.cc:1261] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:59:40.461879 17376 consensus_queue.cc:1035] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:59:40.464398 17376 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:40.467676 6637 catalog_manager.cc:5484] T c8c237c50773494abf8813494be0f184 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 2 leader_uuid: "4f53d80e39be410d92d4766692aff1c6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:40.472556 8903 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:40.475955 7182 raft_consensus.cc:2932] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:40.490262 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.501463 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:40.502816 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:40.508446 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.510725 8751 tablet_service.cc:1514] Processing DeleteTablet for tablet c8c237c50773494abf8813494be0f184 with delete_type TABLET_DATA_TOMBSTONED (TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47850
I20240502 01:59:40.513674 17761 tablet_replica.cc:330] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: stopping tablet replica
I20240502 01:59:40.514110 17761 raft_consensus.cc:2229] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:40.514369 17761 raft_consensus.cc:2258] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:40.515384 17761 ts_tablet_manager.cc:1875] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:40.519380 17761 ts_tablet_manager.cc:1888] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:59:40.519512 17761 log.cc:1196] T c8c237c50773494abf8813494be0f184 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal/wals/c8c237c50773494abf8813494be0f184
I20240502 01:59:40.520702 6627 catalog_manager.cc:4830] TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): tablet c8c237c50773494abf8813494be0f184 (table rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]) successfully deleted
I20240502 01:59:40.529865 6763 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:40.531980 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.551384 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.567778 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:40.584463 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.588786 8883 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.592558 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.609071 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.613323 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:40.614850 6763 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:40.622308 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:40.628640 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.635041 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.636868 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:40.643455 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:40.647269 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:40.673908 6763 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:40.683096 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.689801 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.690594 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:40.724573 8641 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 01:59:40.725034 17184 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:59:40.742094 6783 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f6c07e3923d47f29ef32561d7f438e1"
dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49180
I20240502 01:59:40.742269 6783 raft_consensus.cc:595] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Received request to transfer leadership
I20240502 01:59:40.756266 6783 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 3 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 01:59:40.756999 17184 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:59:40.765095 8509 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 01:59:40.765720 8760 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 3 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 01:59:40.766484 17184 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:59:40.766808 17184 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:59:40.774925 17110 raft_consensus.cc:979] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e: : Instructing follower 070558c278e0457b9cc070ac0d662b5d to start an election
I20240502 01:59:40.775043 17110 raft_consensus.cc:1067] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Signalling peer 070558c278e0457b9cc070ac0d662b5d to start an election
I20240502 01:59:40.775648 8903 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f6c07e3923d47f29ef32561d7f438e1"
dest_uuid: "070558c278e0457b9cc070ac0d662b5d"
from {username='slave'} at 127.27.111.193:48217
I20240502 01:59:40.775852 8903 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:40.775969 8903 raft_consensus.cc:3037] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:40.777091 8903 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:40.778738 6916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6c07e3923d47f29ef32561d7f438e1" candidate_uuid: "070558c278e0457b9cc070ac0d662b5d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:40.778888 6916 raft_consensus.cc:3037] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:40.780025 6916 raft_consensus.cc:2454] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 070558c278e0457b9cc070ac0d662b5d in term 2.
I20240502 01:59:40.781239 6783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6c07e3923d47f29ef32561d7f438e1" candidate_uuid: "070558c278e0457b9cc070ac0d662b5d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:40.781378 6783 raft_consensus.cc:3032] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:40.781424 6783 raft_consensus.cc:729] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:40.781582 6783 consensus_queue.cc:260] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:40.781813 6783 raft_consensus.cc:3037] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:40.782981 6783 raft_consensus.cc:2454] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 070558c278e0457b9cc070ac0d662b5d in term 2.
I20240502 01:59:40.783663 8839 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 070558c278e0457b9cc070ac0d662b5d, 85e78945910a4e1abfa101d2edc3644e, e7bcff2fef614e7a9981554c9e6db3c8; no voters:
I20240502 01:59:40.783949 17727 raft_consensus.cc:2781] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:40.784121 17727 raft_consensus.cc:686] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [term 2 LEADER]: Becoming Leader. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Running, Role: LEADER
I20240502 01:59:40.784412 17727 consensus_queue.cc:237] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:40.786029 8903 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [CANDIDATE]: Term 2 election: Requested vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
I20240502 01:59:40.786542 8641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6c07e3923d47f29ef32561d7f438e1" candidate_uuid: "070558c278e0457b9cc070ac0d662b5d" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "4f53d80e39be410d92d4766692aff1c6"
I20240502 01:59:40.786674 8641 raft_consensus.cc:3037] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:40.786761 6635 catalog_manager.cc:5484] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d reported cstate change: term changed from 1 to 2, leader changed from 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) to 070558c278e0457b9cc070ac0d662b5d (127.27.111.200). New cstate: current_term: 2 leader_uuid: "070558c278e0457b9cc070ac0d662b5d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:40.787665 8641 raft_consensus.cc:2454] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 070558c278e0457b9cc070ac0d662b5d in term 2.
I20240502 01:59:40.815569 8883 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:40.826700 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:40.839632 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:40.840425 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.840917 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.844715 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:40.851233 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:40.888908 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.904858 8883 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:40.907130 7048 consensus_queue.cc:237] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:40.908371 8760 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:40.909461 16110 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:40.911218 16110 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:40.913594 8760 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:40.916852 6640 catalog_manager.cc:5484] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 5 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 2 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:40.919974 7048 consensus_queue.cc:237] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:40.920728 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet a0548274c47148d9b611fd3c978a0c34 with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55860
W20240502 01:59:40.925748 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0548274c47148d9b611fd3c978a0c34. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:40.926334 8760 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:59:40.927801 17289 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:40.933596 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.933951 16110 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:40.937944 8760 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:40.941290 17956 tablet_replica.cc:330] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:40.941509 17956 raft_consensus.cc:2229] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:40.941622 6642 catalog_manager.cc:5484] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 reported cstate change: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 2 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:40.944731 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a0548274c47148d9b611fd3c978a0c34 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:59:40.946883 17956 raft_consensus.cc:2258] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:40.949556 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:40.951880 17956 ts_tablet_manager.cc:1875] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:40.956357 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:40.958673 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:40.959537 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.973178 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:40.975908 8883 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:40.989624 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:40.993113 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:41.001940 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:41.003439 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.005571 17956 ts_tablet_manager.cc:1888] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:59:41.005743 17956 log.cc:1196] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/a0548274c47148d9b611fd3c978a0c34
I20240502 01:59:41.006364 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet a0548274c47148d9b611fd3c978a0c34 (table rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]) successfully deleted
I20240502 01:59:41.009439 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.019655 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.060266 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:41.060544 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.117551 8883 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.124262 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:41.131438 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:41.134986 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.135491 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.143030 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:41.156445 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:41.173758 7048 consensus_queue.cc:237] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:41.175498 7182 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:59:41.175823 8509 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:59:41.176105 16110 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:59:41.176270 17289 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:59:41.178681 8641 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:59:41.179126 16110 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:59:41.185256 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.201445 16110 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:41.202883 7182 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:41.203936 8509 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:41.206568 6642 catalog_manager.cc:5484] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:41.207515 8641 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:41.216912 8751 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a276e9780e448a9b3c6eab67925514 with delete_type TABLET_DATA_TOMBSTONED (TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47850
I20240502 01:59:41.219087 18064 tablet_replica.cc:330] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: stopping tablet replica
I20240502 01:59:41.219331 18064 raft_consensus.cc:2229] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:41.219532 18064 raft_consensus.cc:2258] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:41.220288 18064 ts_tablet_manager.cc:1875] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:41.227948 6783 raft_consensus.cc:1261] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Refusing update from remote peer 070558c278e0457b9cc070ac0d662b5d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:41.228513 17882 consensus_queue.cc:1035] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:41.235534 7182 consensus_queue.cc:237] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:41.236280 18064 ts_tablet_manager.cc:1888] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:59:41.236382 18064 log.cc:1196] T 47a276e9780e448a9b3c6eab67925514 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-6/wal/wals/47a276e9780e448a9b3c6eab67925514
I20240502 01:59:41.237335 8760 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 3 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 01:59:41.237526 6627 catalog_manager.cc:4830] TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): tablet 47a276e9780e448a9b3c6eab67925514 (table rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]) successfully deleted
I20240502 01:59:41.238060 8509 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 01:59:41.238260 8641 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 01:59:41.238983 17641 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:59:41.239161 17866 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:59:41.239322 17641 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:59:41.241833 8509 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:41.241487 17865 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:41.242592 8641 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:41.245079 6640 catalog_manager.cc:5484] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c reported cstate change: config changed from index 10 to 12, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 3 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:41.246565 8760 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:41.251303 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet e6ac26fa6de44f7ca68e6c0b6b959db7 with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:41.251853 17956 tablet_replica.cc:330] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:41.252004 17956 raft_consensus.cc:2229] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:41.252116 17956 raft_consensus.cc:2258] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:41.253355 17956 ts_tablet_manager.cc:1875] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:41.255728 17956 ts_tablet_manager.cc:1888] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240502 01:59:41.255797 17956 log.cc:1196] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/e6ac26fa6de44f7ca68e6c0b6b959db7
I20240502 01:59:41.256553 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet e6ac26fa6de44f7ca68e6c0b6b959db7 (table rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]) successfully deleted
I20240502 01:59:41.311147 6917 raft_consensus.cc:1261] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Refusing update from remote peer 070558c278e0457b9cc070ac0d662b5d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:41.312263 17882 consensus_queue.cc:1035] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:41.320364 8641 raft_consensus.cc:1261] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Refusing update from remote peer 070558c278e0457b9cc070ac0d662b5d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:41.321496 17882 consensus_queue.cc:1035] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:41.326162 18183 ts_tablet_manager.cc:902] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:41.326937 18183 tablet_copy_client.cc:308] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:41.327481 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet a0548274c47148d9b611fd3c978a0c34 from peer 070558c278e0457b9cc070ac0d662b5d ({username='slave'} at 127.27.111.200:49029)
I20240502 01:59:41.327585 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet a0548274c47148d9b611fd3c978a0c34 from peer 070558c278e0457b9cc070ac0d662b5d at {username='slave'} at 127.27.111.200:49029: session id = 070558c278e0457b9cc070ac0d662b5d-a0548274c47148d9b611fd3c978a0c34
I20240502 01:59:41.328694 7070 tablet_copy_source_session.cc:205] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:41.330291 18183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0548274c47148d9b611fd3c978a0c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:41.340807 18183 tablet_copy_client.cc:783] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:41.341138 18183 tablet_copy_client.cc:647] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:41.346194 18183 tablet_copy_client.cc:523] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:41.351615 18183 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap starting.
I20240502 01:59:41.402526 18183 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:41.404155 18183 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d: Bootstrap complete.
I20240502 01:59:41.405920 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:41.417315 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:41.419942 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:41.412205 8883 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.412324 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.421509 18183 ts_tablet_manager.cc:1367] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d: Time spent bootstrapping tablet: real 0.070s user 0.031s sys 0.000s
I20240502 01:59:41.423027 18183 raft_consensus.cc:348] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:41.432304 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:41.440274 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.440865 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.453526 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.456152 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.460644 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:41.465032 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:41.453030 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.469544 18183 raft_consensus.cc:729] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070558c278e0457b9cc070ac0d662b5d, State: Initialized, Role: FOLLOWER
I20240502 01:59:41.469789 18183 consensus_queue.cc:260] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:41.470191 18183 ts_tablet_manager.cc:1398] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d: Time spent starting tablet: real 0.048s user 0.001s sys 0.000s
I20240502 01:59:41.471127 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 070558c278e0457b9cc070ac0d662b5d-a0548274c47148d9b611fd3c978a0c34 received from {username='slave'} at 127.27.111.200:49029
I20240502 01:59:41.471207 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 070558c278e0457b9cc070ac0d662b5d-a0548274c47148d9b611fd3c978a0c34 on tablet a0548274c47148d9b611fd3c978a0c34 with peer 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:41.473938 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:41.477937 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.484387 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:41.505062 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.507335 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.511508 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:41.538952 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:41.547355 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:41.550859 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:41.551018 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.559983 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:41.562168 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:41.579684 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:41.596307 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:41.618664 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.622718 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.636231 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.639580 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.660693 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.666306 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.668195 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:41.668534 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:41.683007 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.691649 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:41.692441 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.721022 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:41.723260 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.804700 6917 consensus_queue.cc:237] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:41.806600 8904 raft_consensus.cc:1261] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:59:41.807040 8509 raft_consensus.cc:1261] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:59:41.807088 17106 consensus_queue.cc:1035] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:41.807972 17106 consensus_queue.cc:1035] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:41.809346 7048 consensus_queue.cc:237] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:41.810637 17106 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:41.811640 8760 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:59:41.812047 8641 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:59:41.813141 8508 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:41.814882 8509 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:59:41.815623 18057 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:41.815811 6641 catalog_manager.cc:5484] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:41.815999 18057 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:41.816258 18057 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:41.818037 8904 raft_consensus.cc:2932] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 01:59:41.819921 17289 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:41.821259 8641 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:41.823967 8509 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:41.824841 6635 catalog_manager.cc:5484] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 7 to 8, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:41.826200 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:41.826930 8760 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:41.829649 18377 tablet_replica.cc:330] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:41.829833 18377 raft_consensus.cc:2229] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:41.829955 18377 raft_consensus.cc:2258] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:41.830567 18377 ts_tablet_manager.cc:1875] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:41.833369 18377 ts_tablet_manager.cc:1888] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:59:41.833503 18377 log.cc:1196] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/9ecbf1c8ee8c46b195d2374a5db81ac0
I20240502 01:59:41.834133 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet 9ecbf1c8ee8c46b195d2374a5db81ac0 (table rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]) successfully deleted
I20240502 01:59:41.837005 7182 consensus_queue.cc:237] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:41.838598 6783 raft_consensus.cc:1261] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:41.838599 8904 raft_consensus.cc:1261] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:41.839136 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 48eeb721a60c4be888041bd31cfeab7a with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:41.839118 17641 consensus_queue.cc:1035] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:41.839354 18377 tablet_replica.cc:330] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:41.839537 18377 raft_consensus.cc:2229] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:41.839699 17641 consensus_queue.cc:1035] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:41.839833 18377 raft_consensus.cc:2258] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:41.840945 18377 ts_tablet_manager.cc:1875] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:41.841498 17641 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:41.843909 18377 ts_tablet_manager.cc:1888] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 01:59:41.844628 6636 catalog_manager.cc:5484] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 6 to 7, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:41.845619 7050 consensus_queue.cc:237] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:41.846959 7182 raft_consensus.cc:1261] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:41.847141 8509 raft_consensus.cc:1261] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:41.844370 6785 consensus_queue.cc:237] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:41.847892 18375 consensus_queue.cc:1035] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:41.848242 18375 consensus_queue.cc:1035] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:41.845218 6783 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:41.849877 8903 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:59:41.850275 8641 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:59:41.850859 17110 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:41.851202 17110 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:41.851451 8903 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
W20240502 01:59:41.852360 6721 consensus_peers.cc:485] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 034cf1c3d47a4f7785c16d16815c229d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:41.853173 18377 log.cc:1196] T 48eeb721a60c4be888041bd31cfeab7a P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/48eeb721a60c4be888041bd31cfeab7a
I20240502 01:59:41.853137 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fa358140eb742b587a4d6a203738b4f with delete_type TABLET_DATA_TOMBSTONED (TS e7bcff2fef614e7a9981554c9e6db3c8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59160
I20240502 01:59:41.853535 18414 tablet_replica.cc:330] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: stopping tablet replica
I20240502 01:59:41.853724 18414 raft_consensus.cc:2229] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:41.853910 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet 48eeb721a60c4be888041bd31cfeab7a (table rebalance_test_table_5 [id=bfc1490f5eee4d0bb5455e4b3ee959f1]) successfully deleted
I20240502 01:59:41.854270 18414 raft_consensus.cc:2258] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:41.854935 18414 ts_tablet_manager.cc:1875] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:41.855443 18364 raft_consensus.cc:2932] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:41.856505 8509 raft_consensus.cc:2932] T d4d429fe60e74108bc0c9ca8509365f2 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:41.857795 18414 ts_tablet_manager.cc:1888] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:59:41.857920 18414 log.cc:1196] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/7fa358140eb742b587a4d6a203738b4f
I20240502 01:59:41.857904 7182 raft_consensus.cc:2932] T d4d429fe60e74108bc0c9ca8509365f2 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:41.859086 6630 catalog_manager.cc:4830] TS e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): tablet 7fa358140eb742b587a4d6a203738b4f (table rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]) successfully deleted
I20240502 01:59:41.859129 6640 catalog_manager.cc:5484] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 6 to 7, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:41.863951 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d429fe60e74108bc0c9ca8509365f2 with delete_type TABLET_DATA_TOMBSTONED (TS e7bcff2fef614e7a9981554c9e6db3c8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59160
I20240502 01:59:41.864193 18414 tablet_replica.cc:330] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8: stopping tablet replica
I20240502 01:59:41.864351 18414 raft_consensus.cc:2229] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:41.864527 18414 raft_consensus.cc:2258] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:41.865078 18414 ts_tablet_manager.cc:1875] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:41.866324 8904 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:41.865728 18409 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:41.867266 18414 ts_tablet_manager.cc:1888] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:59:41.867748 18414 log.cc:1196] T d4d429fe60e74108bc0c9ca8509365f2 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/d4d429fe60e74108bc0c9ca8509365f2
I20240502 01:59:41.868269 6630 catalog_manager.cc:4830] TS e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): tablet d4d429fe60e74108bc0c9ca8509365f2 (table rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]) successfully deleted
I20240502 01:59:41.869794 8641 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:41.869933 6639 catalog_manager.cc:5484] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d reported cstate change: config changed from index 8 to 9, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 2 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:41.927232 8904 raft_consensus.cc:1203] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 01:59:41.992513 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:41.993526 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:41.999024 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.004864 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.024334 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.027776 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:42.032588 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:42.040668 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:42.041424 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.059072 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.059252 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:42.059419 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:42.063786 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.071821 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.075874 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:42.082995 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.092487 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.101888 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.103524 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:42.118754 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:42.135082 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:42.152451 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:42.187379 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.213266 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.227995 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.235143 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.235425 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:42.261144 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.279825 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:42.298676 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:42.309170 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.313004 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:42.316629 18637 ts_tablet_manager.cc:902] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 01:59:42.317114 18637 tablet_copy_client.cc:308] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 01:59:42.317584 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet 034cf1c3d47a4f7785c16d16815c229d from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:33267)
I20240502 01:59:42.317687 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet 034cf1c3d47a4f7785c16d16815c229d from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:33267: session id = 954cd49326bf405abd81182d75dc783c-034cf1c3d47a4f7785c16d16815c229d
I20240502 01:59:42.318717 6805 tablet_copy_source_session.cc:205] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:42.319653 18637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 034cf1c3d47a4f7785c16d16815c229d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:42.322332 18637 tablet_copy_client.cc:783] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:42.322540 18637 tablet_copy_client.cc:647] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:42.332193 6785 consensus_queue.cc:237] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:42.333140 18637 tablet_copy_client.cc:523] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:42.334026 8509 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:42.334458 6917 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:42.334687 18409 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:42.335707 18637 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 01:59:42.337695 18409 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:42.360074 18408 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:42.365187 8509 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:42.365701 6643 catalog_manager.cc:5484] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 6 to 7, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:42.388146 6917 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:42.391353 8884 tablet_service.cc:1514] Processing DeleteTablet for tablet 0359dfbfd11746f1bf62a465f14634ec with delete_type TABLET_DATA_TOMBSTONED (TS 070558c278e0457b9cc070ac0d662b5d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35238
I20240502 01:59:42.391958 18692 tablet_replica.cc:330] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d: stopping tablet replica
I20240502 01:59:42.392346 18692 raft_consensus.cc:2229] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:42.392603 18692 raft_consensus.cc:2258] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:42.394992 18692 ts_tablet_manager.cc:1875] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:42.397534 18692 ts_tablet_manager.cc:1888] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:59:42.397722 18692 log.cc:1196] T 0359dfbfd11746f1bf62a465f14634ec P 070558c278e0457b9cc070ac0d662b5d: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal/wals/0359dfbfd11746f1bf62a465f14634ec
I20240502 01:59:42.398384 6627 catalog_manager.cc:4830] TS 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): tablet 0359dfbfd11746f1bf62a465f14634ec (table rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]) successfully deleted
I20240502 01:59:42.402081 7050 consensus_queue.cc:237] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:42.403570 7182 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:59:42.403759 8641 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:59:42.404536 18375 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:59:42.405411 18057 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:59:42.407358 7050 consensus_queue.cc:237] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:42.409040 8757 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240502 01:59:42.409145 6984 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): Couldn't send request to peer 85e78945910a4e1abfa101d2edc3644e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:42.409503 18375 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:42.409682 8904 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:59:42.410135 18375 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:59:42.416790 7050 consensus_queue.cc:237] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 01:59:42.418573 7181 raft_consensus.cc:1261] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:42.418604 8509 raft_consensus.cc:1261] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:42.419071 18375 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:42.419703 18718 consensus_queue.cc:1035] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:42.421742 8757 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 4e6e76b381af4cdc90bd3e9e6e8b5cf2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:42.422019 8904 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:42.423715 18057 consensus_queue.cc:1035] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:42.425333 6643 catalog_manager.cc:5484] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New cstate: current_term: 2 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 01:59:42.430845 18718 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:42.432102 18721 raft_consensus.cc:2932] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:42.434623 8641 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:42.435261 6641 catalog_manager.cc:5484] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 10 to 11, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:42.435606 18637 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:42.436028 18637 tablet_bootstrap.cc:492] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
I20240502 01:59:42.438097 7182 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:42.438455 8509 raft_consensus.cc:2932] T 9313079c48c94ab1b7e022a91befafcd P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:42.439849 7181 raft_consensus.cc:2932] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:42.444795 18637 ts_tablet_manager.cc:1367] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.109s user 0.017s sys 0.017s
I20240502 01:59:42.445367 18637 raft_consensus.cc:348] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:42.445627 18637 raft_consensus.cc:729] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 01:59:42.445830 18637 consensus_queue.cc:260] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:42.446288 18637 ts_tablet_manager.cc:1398] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:59:42.446326 6634 catalog_manager.cc:5484] T 9313079c48c94ab1b7e022a91befafcd P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 6 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 01:59:42.446866 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-034cf1c3d47a4f7785c16d16815c229d received from {username='slave'} at 127.27.111.197:33267
I20240502 01:59:42.446964 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 954cd49326bf405abd81182d75dc783c-034cf1c3d47a4f7785c16d16815c229d on tablet 034cf1c3d47a4f7785c16d16815c229d with peer 954cd49326bf405abd81182d75dc783c
I20240502 01:59:42.449767 8489 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a276e9780e448a9b3c6eab67925514 with delete_type TABLET_DATA_TOMBSTONED (TS 954cd49326bf405abd81182d75dc783c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:46610
I20240502 01:59:42.450671 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 9313079c48c94ab1b7e022a91befafcd with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:42.457166 18745 tablet_replica.cc:330] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:42.458031 18745 raft_consensus.cc:2229] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:42.459004 18745 raft_consensus.cc:2258] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:42.459672 18745 ts_tablet_manager.cc:1875] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:42.464393 18745 ts_tablet_manager.cc:1888] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:59:42.464571 18745 log.cc:1196] T 9313079c48c94ab1b7e022a91befafcd P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/9313079c48c94ab1b7e022a91befafcd
I20240502 01:59:42.465217 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet 9313079c48c94ab1b7e022a91befafcd (table rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]) successfully deleted
I20240502 01:59:42.468119 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.470631 18744 tablet_replica.cc:330] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c: stopping tablet replica
I20240502 01:59:42.470994 18744 raft_consensus.cc:2229] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:42.471273 18744 raft_consensus.cc:2258] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:42.473753 18744 ts_tablet_manager.cc:1875] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:42.476632 18744 ts_tablet_manager.cc:1888] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 01:59:42.476730 18744 log.cc:1196] T 47a276e9780e448a9b3c6eab67925514 P 954cd49326bf405abd81182d75dc783c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal/wals/47a276e9780e448a9b3c6eab67925514
I20240502 01:59:42.477365 6630 catalog_manager.cc:4830] TS 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): tablet 47a276e9780e448a9b3c6eab67925514 (table rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]) successfully deleted
I20240502 01:59:42.478614 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:42.488610 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.489485 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.489605 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.508857 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:42.515092 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:42.515457 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:42.595850 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.605305 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:42.607368 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:42.618988 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:42.628334 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.631203 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.639851 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.640360 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:42.715556 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.728850 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:42.736109 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:42.739259 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:42.748888 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:42.752542 6894 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.755256 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:42.758689 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.760336 6917 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44c55b199a24432e902cce9121fc0780"
dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35984
I20240502 01:59:42.760495 6917 raft_consensus.cc:595] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:59:42.767645 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:42.778780 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.779270 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.794243 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:42.794539 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.795168 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:42.800650 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.806007 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:42.833556 6763 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:59:42.835458 8621 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:59:42.838654 7161 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.847919 17106 raft_consensus.cc:979] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8: : Instructing follower 86bc8001c3464b929cd5c873861baa0c to start an election
I20240502 01:59:42.848037 17106 raft_consensus.cc:1067] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Signalling peer 86bc8001c3464b929cd5c873861baa0c to start an election
I20240502 01:59:42.848546 7181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44c55b199a24432e902cce9121fc0780"
dest_uuid: "86bc8001c3464b929cd5c873861baa0c"
from {username='slave'} at 127.27.111.194:34149
I20240502 01:59:42.848764 7181 raft_consensus.cc:482] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:42.848876 7181 raft_consensus.cc:3037] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:42.850049 7181 raft_consensus.cc:504] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:42.850384 8509 raft_consensus.cc:1203] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240502 01:59:42.851164 8904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c55b199a24432e902cce9121fc0780" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "070558c278e0457b9cc070ac0d662b5d"
I20240502 01:59:42.851348 8904 raft_consensus.cc:3037] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:42.853219 8904 raft_consensus.cc:2454] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 2.
I20240502 01:59:42.853235 6917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c55b199a24432e902cce9121fc0780" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "e7bcff2fef614e7a9981554c9e6db3c8"
I20240502 01:59:42.853355 6917 raft_consensus.cc:3032] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:59:42.853395 6917 raft_consensus.cc:729] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Running, Role: LEADER
I20240502 01:59:42.853544 6917 consensus_queue.cc:260] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:42.853742 6917 raft_consensus.cc:3037] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:42.853953 7181 leader_election.cc:288] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 2 election: Requested vote from peers e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767)
I20240502 01:59:42.854784 8509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c55b199a24432e902cce9121fc0780" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "954cd49326bf405abd81182d75dc783c"
I20240502 01:59:42.854918 8509 raft_consensus.cc:3037] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:59:42.855855 8509 raft_consensus.cc:2454] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 2.
I20240502 01:59:42.856168 7117 leader_election.cc:302] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 070558c278e0457b9cc070ac0d662b5d, 86bc8001c3464b929cd5c873861baa0c, 954cd49326bf405abd81182d75dc783c; no voters:
I20240502 01:59:42.856344 18738 raft_consensus.cc:2781] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:59:42.856353 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.856428 18738 raft_consensus.cc:686] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 2 LEADER]: Becoming Leader. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Running, Role: LEADER
I20240502 01:59:42.856562 18738 consensus_queue.cc:237] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:42.858467 6917 raft_consensus.cc:2454] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 2.
I20240502 01:59:42.858889 6634 catalog_manager.cc:5484] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: term changed from 1 to 2, leader changed from e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) to 86bc8001c3464b929cd5c873861baa0c (127.27.111.196). New cstate: current_term: 2 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:42.868764 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:42.885990 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:42.892370 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.893870 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:42.931164 6785 consensus_queue.cc:237] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:42.933548 8509 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:59:42.933566 6917 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 01:59:42.934043 6718 consensus_peers.cc:485] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0359dfbfd11746f1bf62a465f14634ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:42.934335 18409 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:42.934537 18408 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:42.936928 18409 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:42.941042 6634 catalog_manager.cc:5484] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 7 to 8, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:42.944377 8509 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:42.944764 6917 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:42.955742 18969 ts_tablet_manager.cc:902] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 01:59:42.956692 18969 tablet_copy_client.cc:308] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 01:59:42.957784 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet a0548274c47148d9b611fd3c978a0c34 from peer 85e78945910a4e1abfa101d2edc3644e ({username='slave'} at 127.27.111.193:42103)
I20240502 01:59:42.957917 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet a0548274c47148d9b611fd3c978a0c34 from peer 85e78945910a4e1abfa101d2edc3644e at {username='slave'} at 127.27.111.193:42103: session id = 85e78945910a4e1abfa101d2edc3644e-a0548274c47148d9b611fd3c978a0c34
I20240502 01:59:42.959033 7070 tablet_copy_source_session.cc:205] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:59:42.960924 18969 ts_tablet_manager.cc:1875] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:59:42.967193 18969 ts_tablet_manager.cc:1888] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240502 01:59:42.967406 18969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0548274c47148d9b611fd3c978a0c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:59:42.968063 6785 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "034cf1c3d47a4f7785c16d16815c229d"
dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58338
I20240502 01:59:42.968200 6785 raft_consensus.cc:595] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Received request to transfer leadership
I20240502 01:59:42.968959 18969 tablet_copy_client.cc:783] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 0 data blocks...
I20240502 01:59:42.969120 18969 tablet_copy_client.cc:647] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 1 WAL segments...
I20240502 01:59:42.972503 18969 tablet_copy_client.cc:523] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:59:42.974342 18969 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 01:59:42.987239 18961 raft_consensus.cc:979] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e: : Instructing follower 4f53d80e39be410d92d4766692aff1c6 to start an election
I20240502 01:59:42.987373 18961 raft_consensus.cc:1067] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Signalling peer 4f53d80e39be410d92d4766692aff1c6 to start an election
I20240502 01:59:42.988018 8641 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "034cf1c3d47a4f7785c16d16815c229d"
dest_uuid: "4f53d80e39be410d92d4766692aff1c6"
from {username='slave'} at 127.27.111.193:36281
I20240502 01:59:42.988152 8641 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:59:42.988219 8641 raft_consensus.cc:3037] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:42.989287 8641 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:42.989830 8641 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [CANDIDATE]: Term 3 election: Requested vote from peers 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767)
I20240502 01:59:42.990685 8904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034cf1c3d47a4f7785c16d16815c229d" candidate_uuid: "4f53d80e39be410d92d4766692aff1c6" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "070558c278e0457b9cc070ac0d662b5d"
I20240502 01:59:42.990808 8904 raft_consensus.cc:3037] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:42.991844 8904 raft_consensus.cc:2454] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f53d80e39be410d92d4766692aff1c6 in term 3.
I20240502 01:59:42.992872 6785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034cf1c3d47a4f7785c16d16815c229d" candidate_uuid: "4f53d80e39be410d92d4766692aff1c6" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "85e78945910a4e1abfa101d2edc3644e"
I20240502 01:59:42.993006 6785 raft_consensus.cc:3032] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:59:42.993050 6785 raft_consensus.cc:729] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Running, Role: LEADER
I20240502 01:59:42.993186 6785 consensus_queue.cc:260] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:42.993356 6785 raft_consensus.cc:3037] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:42.994356 6785 raft_consensus.cc:2454] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f53d80e39be410d92d4766692aff1c6 in term 3.
I20240502 01:59:42.994992 8576 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6, 85e78945910a4e1abfa101d2edc3644e; no voters:
I20240502 01:59:42.995179 18182 raft_consensus.cc:2781] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:59:42.995322 18182 raft_consensus.cc:686] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 3 LEADER]: Becoming Leader. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Running, Role: LEADER
I20240502 01:59:42.995476 18182 consensus_queue.cc:237] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:43.000640 6635 catalog_manager.cc:5484] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: term changed from 2 to 3, leader changed from 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) to 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198). New cstate: current_term: 3 leader_uuid: "4f53d80e39be410d92d4766692aff1c6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:43.005278 18969 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:59:43.005609 18969 tablet_bootstrap.cc:492] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap complete.
I20240502 01:59:43.005805 18969 ts_tablet_manager.cc:1367] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.000s
I20240502 01:59:43.006253 18969 raft_consensus.cc:348] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:43.006385 18969 raft_consensus.cc:729] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 01:59:43.006572 18969 consensus_queue.cc:260] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 01:59:43.006861 18969 ts_tablet_manager.cc:1398] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 01:59:43.007501 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 85e78945910a4e1abfa101d2edc3644e-a0548274c47148d9b611fd3c978a0c34 received from {username='slave'} at 127.27.111.193:42103
I20240502 01:59:43.007582 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 85e78945910a4e1abfa101d2edc3644e-a0548274c47148d9b611fd3c978a0c34 on tablet a0548274c47148d9b611fd3c978a0c34 with peer 85e78945910a4e1abfa101d2edc3644e
I20240502 01:59:43.012310 8509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "034cf1c3d47a4f7785c16d16815c229d" candidate_uuid: "4f53d80e39be410d92d4766692aff1c6" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "954cd49326bf405abd81182d75dc783c"
I20240502 01:59:43.012475 8509 raft_consensus.cc:3037] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:59:43.013604 8509 raft_consensus.cc:2454] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f53d80e39be410d92d4766692aff1c6 in term 3.
I20240502 01:59:43.029184 8509 consensus_queue.cc:237] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:43.030936 8641 raft_consensus.cc:1261] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Refusing update from remote peer 954cd49326bf405abd81182d75dc783c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:59:43.032022 18218 consensus_queue.cc:1035] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:43.033013 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.034896 7050 raft_consensus.cc:1261] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Refusing update from remote peer 954cd49326bf405abd81182d75dc783c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:59:43.036424 18218 consensus_queue.cc:1035] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:43.045017 18218 raft_consensus.cc:2932] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:43.047286 8641 raft_consensus.cc:2932] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:43.050599 6635 catalog_manager.cc:5484] T 17235e019e644b258266636fb6be6648 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 6 to 8, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New cstate: current_term: 2 leader_uuid: "954cd49326bf405abd81182d75dc783c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:43.053035 7050 raft_consensus.cc:2932] T 17235e019e644b258266636fb6be6648 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:43.057561 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.058050 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet 17235e019e644b258266636fb6be6648 with delete_type TABLET_DATA_TOMBSTONED (TS e7bcff2fef614e7a9981554c9e6db3c8 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59160
I20240502 01:59:43.059156 6785 consensus_queue.cc:237] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:43.060446 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:43.061741 6917 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:59:43.062269 18961 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 01:59:43.063004 6718 consensus_peers.cc:485] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0359dfbfd11746f1bf62a465f14634ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:43.064342 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:43.065871 18962 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:43.068208 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:43.068626 6917 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:43.068866 6637 catalog_manager.cc:5484] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 8 to 9, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) evicted. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } }
I20240502 01:59:43.072903 8489 tablet_service.cc:1514] Processing DeleteTablet for tablet 0359dfbfd11746f1bf62a465f14634ec with delete_type TABLET_DATA_TOMBSTONED (TS 954cd49326bf405abd81182d75dc783c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:46610
I20240502 01:59:43.075155 19069 tablet_replica.cc:330] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8: stopping tablet replica
I20240502 01:59:43.077514 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:43.078336 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:43.082870 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:43.104974 19075 tablet_replica.cc:330] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: stopping tablet replica
I20240502 01:59:43.105235 19075 raft_consensus.cc:2229] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:43.105429 19075 raft_consensus.cc:2258] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:43.106184 19075 ts_tablet_manager.cc:1875] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:43.108821 19075 ts_tablet_manager.cc:1888] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:59:43.108992 19075 log.cc:1196] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal/wals/0359dfbfd11746f1bf62a465f14634ec
I20240502 01:59:43.109622 6630 catalog_manager.cc:4830] TS 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): tablet 0359dfbfd11746f1bf62a465f14634ec (table rebalance_test_table_1 [id=afb5b6283779407b9ddcb26793f68cf5]) successfully deleted
I20240502 01:59:43.119508 19069 raft_consensus.cc:2229] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:43.119716 19069 raft_consensus.cc:2258] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:43.124822 19069 ts_tablet_manager.cc:1875] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:43.143712 19069 ts_tablet_manager.cc:1888] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 01:59:43.148638 19069 log.cc:1196] T 17235e019e644b258266636fb6be6648 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/17235e019e644b258266636fb6be6648
I20240502 01:59:43.149391 6630 catalog_manager.cc:4830] TS e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): tablet 17235e019e644b258266636fb6be6648 (table rebalance_test_table_2 [id=9d0e8c4a95a747b18b0b6422ed894715]) successfully deleted
I20240502 01:59:43.168774 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.180379 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:43.199575 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:43.203879 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:43.212945 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:43.220351 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:43.233343 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.244623 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:43.278579 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:43.278868 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:43.282135 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:43.292518 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.294628 8509 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:43.295073 17865 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:43.307363 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:43.316829 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.317847 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:43.329069 6917 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:43.329535 17865 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:43.342542 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:43.366256 8904 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:59:43.371166 17865 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:43.447875 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:43.450301 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:43.451110 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.460453 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:43.471285 6785 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 3 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 01:59:43.471753 19025 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:59:43.479148 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:43.493090 6783 raft_consensus.cc:1203] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240502 01:59:43.496989 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:43.499473 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:43.500052 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:43.501919 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.514096 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:43.521277 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:43.521559 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:43.534580 8904 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d [term 3 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 01:59:43.535063 19025 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:59:43.536151 19288 ts_tablet_manager.cc:902] T 0359dfbfd11746f1bf62a465f14634ec P 4e6e76b381af4cdc90bd3e9e6e8b5cf2: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
F20240502 01:59:43.536505 19288 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0359dfbfd11746f1bf62a465f14634ec.queue_time_us, tablet-download-pool-0359dfbfd11746f1bf62a465f14634ec.queue_time_us) (tablet-download-pool-14e3c78c65334b52aca79d67c4bf45a2.queue_time_us, tablet-download-pool-14e3c78c65334b52aca79d67c4bf45a2.queue_time_us) (tablet-download-pool-14e3c78c65334b52aca79d67c4bf45a2.run_cpu_time_us, tablet-download-pool-14e3c78c65334b52aca79d67c4bf45a2.run_cpu_time_us) (tablet-download-pool-14e3c78c65334b52aca79d67c4bf45a2.run_wall_time_us, tablet-download-pool-14e3c78c65334b52aca79d67c4bf45a2.run_wall_time_us) (tablet-download-pool-47a276e9780e448a9b3c6eab67925514.queue_time_us, tablet-download-pool-47a276e9780e448a9b3c6eab67925514.queue_time_us) (tablet-download-pool-47a276e9780e448a9b3c6eab67925514.run_cpu_time_us, tablet-download-pool-47a276e9780e448a9b3c6eab67925514.run_cpu_time_us) (tablet-download-pool-47a276e9780e448a9b3c6eab67925514.run_wall_time_us, tablet-download-pool-47a276e9780e448a9b3c6eab67925514.run_wall_time_us) (tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.queue_time_us, tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.queue_time_us) (tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.run_cpu_time_us, tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.run_cpu_time_us) (tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.run_wall_time_us, tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.run_wall_time_us) (tablet-download-pool-818dbc9a8e054eeaa122e7919a311806.queue_time_us, tablet-download-pool-818dbc9a8e054eeaa122e7919a311806.queue_time_us) (tablet-download-pool-818dbc9a8e054eeaa122e7919a311806.run_cpu_time_us, tablet-download-pool-818dbc9a8e054eeaa122e7919a311806.run_cpu_time_us) (tablet-download-pool-818dbc9a8e054eeaa122e7919a311806.run_wall_time_us, tablet-download-pool-818dbc9a8e054eeaa122e7919a311806.run_wall_time_us) (tablet-download-pool-8529a93abd6e4d62bbc0bddcf2fda78a.queue_time_us, tablet-download-pool-8529a93abd6e4d62bbc0bddcf2fda78a.queue_time_us) (tablet-download-pool-8529a93abd6e4d62bbc0bddcf2fda78a.run_cpu_time_us, tablet-download-pool-8529a93abd6e4d62bbc0bddcf2fda78a.run_cpu_time_us) (tablet-download-pool-8529a93abd6e4d62bbc0bddcf2fda78a.run_wall_time_us, tablet-download-pool-8529a93abd6e4d62bbc0bddcf2fda78a.run_wall_time_us) (tablet-download-pool-91db5c1b3d2f4d22b5882a61ea0395c7.queue_time_us, tablet-download-pool-91db5c1b3d2f4d22b5882a61ea0395c7.queue_time_us) (tablet-download-pool-91db5c1b3d2f4d22b5882a61ea0395c7.run_cpu_time_us, tablet-download-pool-91db5c1b3d2f4d22b5882a61ea0395c7.run_cpu_time_us) (tablet-download-pool-91db5c1b3d2f4d22b5882a61ea0395c7.run_wall_time_us, tablet-download-pool-91db5c1b3d2f4d22b5882a61ea0395c7.run_wall_time_us) (tablet-download-pool-a0548274c47148d9b611fd3c978a0c34.queue_time_us, tablet-download-pool-a0548274c47148d9b611fd3c978a0c34.queue_time_us) (tablet-download-pool-a0548274c47148d9b611fd3c978a0c34.run_cpu_time_us, tablet-download-pool-a0548274c47148d9b611fd3c978a0c34.run_cpu_time_us) (tablet-download-pool-a0548274c47148d9b611fd3c978a0c34.run_wall_time_us, tablet-download-pool-a0548274c47148d9b611fd3c978a0c34.run_wall_time_us) (tablet-download-pool-a4fb9af11abc49d496b5c3d8048c22c8.queue_time_us, tablet-download-pool-a4fb9af11abc49d496b5c3d8048c22c8.queue_time_us) (tablet-download-pool-a4fb9af11abc49d496b5c3d8048c22c8.run_cpu_time_us, tablet-download-pool-a4fb9af11abc49d496b5c3d8048c22c8.run_cpu_time_us) (tablet-download-pool-a4fb9af11abc49d496b5c3d8048c22c8.run_wall_time_us, tablet-download-pool-a4fb9af11abc49d496b5c3d8048c22c8.run_wall_time_us) (tablet-download-pool-b69d6e907b3b400f93721981732c006b.queue_time_us, tablet-download-pool-b69d6e907b3b400f93721981732c006b.queue_time_us) (tablet-download-pool-b69d6e907b3b400f93721981732c006b.run_cpu_time_us, tablet-download-pool-b69d6e907b3b400f93721981732c006b.run_cpu_time_us) (tablet-download-pool-b69d6e907b3b400f93721981732c006b.run_wall_time_us, tablet-download-pool-b69d6e907b3b400f93721981732c006b.run_wall_time_us) (tablet-download-pool-c8c237c50773494abf8813494be0f184.queue_time_us, tablet-download-pool-c8c237c50773494abf8813494be0f184.queue_time_us) (tablet-download-pool-c8c237c50773494abf8813494be0f184.run_cpu_time_us, tablet-download-pool-c8c237c50773494abf8813494be0f184.run_cpu_time_us) (tablet-download-pool-c8c237c50773494abf8813494be0f184.run_wall_time_us, tablet-download-pool-c8c237c50773494abf8813494be0f184.run_wall_time_us) (tablet-download-pool-dcc5daef41bc44c4855f2bcb44e1c6d5.queue_time_us, tablet-download-pool-dcc5daef41bc44c4855f2bcb44e1c6d5.queue_time_us) (tablet-download-pool-dcc5daef41bc44c4855f2bcb44e1c6d5.run_cpu_time_us, tablet-download-pool-dcc5daef41bc44c4855f2bcb44e1c6d5.run_cpu_time_us) (tablet-download-pool-dcc5daef41bc44c4855f2bcb44e1c6d5.run_wall_time_us, tablet-download-pool-dcc5daef41bc44c4855f2bcb44e1c6d5.run_wall_time_us) (tablet-download-pool-e0e05f96dbc542eb999c16dcc0620666.queue_time_us, tablet-download-pool-e0e05f96dbc542eb999c16dcc0620666.queue_time_us) (tablet-download-pool-e0e05f96dbc542eb999c16dcc0620666.run_cpu_time_us, tablet-download-pool-e0e05f96dbc542eb999c16dcc0620666.run_cpu_time_us) (tablet-download-pool-e0e05f96dbc542eb999c16dcc0620666.run_wall_time_us, tablet-download-pool-e0e05f96dbc542eb999c16dcc0620666.run_wall_time_us) (tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.queue_time_us, tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.queue_time_us) (tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.run_cpu_time_us, tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.run_cpu_time_us) (tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.run_wall_time_us, tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.run_wall_time_us) (tablet-download-pool-ed7b67af7b1c4255a7c2377570b58e70.queue_time_us, tablet-download-pool-ed7b67af7b1c4255a7c2377570b58e70.queue_time_us) (tablet-download-pool-ed7b67af7b1c4255a7c2377570b58e70.run_cpu_time_us, tablet-download-pool-ed7b67af7b1c4255a7c2377570b58e70.run_cpu_time_us) (tablet-download-pool-ed7b67af7b1c4255a7c2377570b58e70.run_wall_time_us, tablet-download-pool-ed7b67af7b1c4255a7c2377570b58e70.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 01:59:43.538105 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:43.539326 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.546411 8509 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 01:59:43.546849 19025 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:59:43.554042 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.570569 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7face6f15dcd google::LogMessage::Fail() at ??:0
@ 0x7face6f19b93 google::LogMessage::SendToLog() at ??:0
@ 0x7face6f157cc google::LogMessage::Flush() at ??:0
@ 0x7face6f16f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 01:59:43.642287 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7face837f044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7face8363f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240502 01:59:43.650445 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7face8361efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 01:59:43.656623 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.660979 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:43.670571 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.687951 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:43.700120 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:43.703490 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.706554 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:43.721920 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:43.728508 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:43.737020 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
@ 0x7faceaf1477a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 01:59:43.740365 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7faceaf1da45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 01:59:43.759811 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:43.767252 8751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:43.769372 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7faceaf9e76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7faceafb24f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7faceaf9d226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240502 01:59:43.828500 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.835378 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
@ 0x7faceafacc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x561a443d175c std::function<>::operator()() at ??:0
@ 0x7face8366e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 01:59:43.846663 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7face83676ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240502 01:59:43.849558 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7face836903b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x561a443d175c std::function<>::operator()() at ??:0
@ 0x7face8356807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7face8b946db start_thread at ??:0
@ 0x7face60cc71f clone at ??:0
I20240502 01:59:43.871162 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:43.883549 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:43.886211 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:43.899348 8904 consensus_queue.cc:237] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:43.900730 6917 raft_consensus.cc:1261] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Refusing update from remote peer 070558c278e0457b9cc070ac0d662b5d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:59:43.901142 18175 consensus_queue.cc:1035] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:43.901432 8641 raft_consensus.cc:1261] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Refusing update from remote peer 070558c278e0457b9cc070ac0d662b5d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:59:43.902585 18175 consensus_queue.cc:1035] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:43.909503 19201 raft_consensus.cc:2932] T 9f6c07e3923d47f29ef32561d7f438e1 P 070558c278e0457b9cc070ac0d662b5d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:43.909902 6917 raft_consensus.cc:2932] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:43.913379 6638 catalog_manager.cc:5484] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 6 to 8, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 2 leader_uuid: "070558c278e0457b9cc070ac0d662b5d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:43.915726 8641 raft_consensus.cc:2932] T 9f6c07e3923d47f29ef32561d7f438e1 P 4f53d80e39be410d92d4766692aff1c6 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:43.953402 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f6c07e3923d47f29ef32561d7f438e1 with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:43.954100 6785 consensus_queue.cc:237] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:43.955906 8641 raft_consensus.cc:1261] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:43.956323 18961 consensus_queue.cc:1035] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:43.962352 7181 consensus_queue.cc:237] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:43.962886 19472 tablet_replica.cc:330] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:43.963006 19472 raft_consensus.cc:2229] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:43.962850 18961 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:43.963099 19472 raft_consensus.cc:2258] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:43.963752 19472 ts_tablet_manager.cc:1875] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:43.963999 8508 raft_consensus.cc:1261] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:43.964169 6917 raft_consensus.cc:1261] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:59:43.964989 19193 consensus_queue.cc:1035] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:43.965837 8641 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:43.966313 19472 ts_tablet_manager.cc:1888] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 01:59:43.966380 19472 log.cc:1196] T 9f6c07e3923d47f29ef32561d7f438e1 P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/9f6c07e3923d47f29ef32561d7f438e1
I20240502 01:59:43.967145 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet 9f6c07e3923d47f29ef32561d7f438e1 (table rebalance_test_table_3 [id=ed198b9f6450449ca6bd353bdb977c91]) successfully deleted
I20240502 01:59:43.968667 6642 catalog_manager.cc:5484] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 6 to 7, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 01:59:43.972981 19193 consensus_queue.cc:1035] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:59:43.976238 19501 raft_consensus.cc:2932] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:43.976866 8508 raft_consensus.cc:2932] T 1c54b912f802461c8cac092ac3222c87 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:43.977072 6917 raft_consensus.cc:2932] T 1c54b912f802461c8cac092ac3222c87 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:43.980468 6638 catalog_manager.cc:5484] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 6 to 7, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:43.989099 8884 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c54b912f802461c8cac092ac3222c87 with delete_type TABLET_DATA_TOMBSTONED (TS 070558c278e0457b9cc070ac0d662b5d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35238
I20240502 01:59:43.990187 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet a4fb9af11abc49d496b5c3d8048c22c8 with delete_type TABLET_DATA_TOMBSTONED (TS e7bcff2fef614e7a9981554c9e6db3c8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59160
I20240502 01:59:43.990626 19505 tablet_replica.cc:330] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8: stopping tablet replica
I20240502 01:59:43.991348 19505 raft_consensus.cc:2229] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:43.991598 19505 raft_consensus.cc:2258] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:43.992153 19505 ts_tablet_manager.cc:1875] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:43.994967 19505 ts_tablet_manager.cc:1888] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:59:43.995036 19505 log.cc:1196] T a4fb9af11abc49d496b5c3d8048c22c8 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/a4fb9af11abc49d496b5c3d8048c22c8
I20240502 01:59:43.995533 6630 catalog_manager.cc:4830] TS e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): tablet a4fb9af11abc49d496b5c3d8048c22c8 (table rebalance_test_table_7 [id=3f6f56d5459f45faa8baa3e7d3fb95cb]) successfully deleted
I20240502 01:59:44.006963 19504 tablet_replica.cc:330] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d: stopping tablet replica
I20240502 01:59:44.007179 19504 raft_consensus.cc:2229] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:44.007771 19504 raft_consensus.cc:2258] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:44.008442 19504 ts_tablet_manager.cc:1875] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:44.010794 19504 ts_tablet_manager.cc:1888] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:59:44.010859 19504 log.cc:1196] T 1c54b912f802461c8cac092ac3222c87 P 070558c278e0457b9cc070ac0d662b5d: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-7/wal/wals/1c54b912f802461c8cac092ac3222c87
I20240502 01:59:44.011350 6627 catalog_manager.cc:4830] TS 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): tablet 1c54b912f802461c8cac092ac3222c87 (table rebalance_test_table_8 [id=eda9a0cbf0634fa49df0814b3e1ce66c]) successfully deleted
I20240502 01:59:44.050324 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.069046 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:44.073818 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:44.076417 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:44.090698 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:44.119599 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.131954 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:44.183149 6982 connection.cc:535] client connection to 127.27.111.199:41741 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:59:44.183318 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:59:44.184298 6849 connection.cc:535] client connection to 127.27.111.199:41741 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:59:44.184448 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:59:44.184468 7114 connection.cc:535] client connection to 127.27.111.199:41741 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:59:44.184594 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:59:44.186194 6718 connection.cc:535] client connection to 127.27.111.199:41741 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:59:44.186331 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:59:44.199088 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:59:44.199185 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:59:44.209022 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:59:44.209120 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:59:44.210024 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:59:44.210120 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:59:44.210166 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:59:44.210724 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 01:59:44.210805 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:59:44.239553 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:44.245827 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.252038 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:44.263283 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:44.263516 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:44.275909 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.277607 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:44.295046 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:44.311741 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 01:59:44.345969 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:44.375689 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.376420 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.377215 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:44.382488 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:44.397316 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:59:44.400362 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:59:44.420733 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:44.425516 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:44.435060 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.435777 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:44.439218 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:44.456329 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:44.457772 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.463685 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:44.476830 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:44.478243 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:44.491716 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:44.493232 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:44.501827 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:44.505543 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:44.514678 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:44.514842 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:44.522522 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.592751 7181 consensus_queue.cc:237] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:44.596042 8508 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:59:44.597443 19501 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:44.600746 8904 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:59:44.601851 19501 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:59:44.606781 8954 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task43FDSt/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 01:59:24.545903 8961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:24.546211 8961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:24.551417 8961 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:59:25.640074 8961 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:59:26.227964 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.228130 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.228215 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.228302 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.228348 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.228420 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.228477 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.228538 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.228583 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.228627 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.228672 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.228715 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.228765 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.229422 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:26.305392 8961 rebalancer_tool.cc:1074] tablet ed7b67af7b1c4255a7c2377570b58e70: 'c6101231591a458384f4ea6d1f0e0ae5' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move scheduled
I20240502 01:59:26.390420 8961 rebalancer_tool.cc:1074] tablet 9ecbf1c8ee8c46b195d2374a5db81ac0: '86bc8001c3464b929cd5c873861baa0c' -> '954cd49326bf405abd81182d75dc783c' move scheduled
I20240502 01:59:26.470815 8961 rebalancer_tool.cc:1074] tablet 39d9e48f67f34ebe850fc388ecd94ca3: '85e78945910a4e1abfa101d2edc3644e' -> '070558c278e0457b9cc070ac0d662b5d' move scheduled
I20240502 01:59:26.625535 8961 rebalancer_tool.cc:1074] tablet 14e3c78c65334b52aca79d67c4bf45a2: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4f53d80e39be410d92d4766692aff1c6' move scheduled
I20240502 01:59:26.689278 8961 rebalancer_tool.cc:1084] tablet 9f6c07e3923d47f29ef32561d7f438e1: 'c6101231591a458384f4ea6d1f0e0ae5' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 01:59:26.689385 8961 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:27.229132 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.229332 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.231040 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.231606 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.231698 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.231793 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.231873 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.232030 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.232110 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.232175 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.232249 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.232460 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.232535 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.232591 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.232635 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.232687 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.232733 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.232784 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.233023 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:27.525357 8961 rebalancer_tool.cc:1074] tablet 48eeb721a60c4be888041bd31cfeab7a: '86bc8001c3464b929cd5c873861baa0c' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move scheduled
I20240502 01:59:27.637125 8961 rebalancer_tool.cc:1074] tablet 91db5c1b3d2f4d22b5882a61ea0395c7: '85e78945910a4e1abfa101d2edc3644e' -> '4f53d80e39be410d92d4766692aff1c6' move scheduled
I20240502 01:59:27.684948 8961 rebalancer_tool.cc:1084] tablet e6ac26fa6de44f7ca68e6c0b6b959db7: 'c6101231591a458384f4ea6d1f0e0ae5' -> '954cd49326bf405abd81182d75dc783c' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240502 01:59:27.685021 8961 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:28.072546 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.072652 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.072739 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.072785 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.072824 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:28.072942 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:28.072978 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.073004 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.073037 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.073066 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:28.073135 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:28.073173 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.073207 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:28.073238 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:28.112289 8961 rebalancer_tool.cc:1074] tablet 8529a93abd6e4d62bbc0bddcf2fda78a: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:28.300158 8961 rebalancer_tool.cc:1074] tablet 7fa358140eb742b587a4d6a203738b4f: 'c6101231591a458384f4ea6d1f0e0ae5' -> '070558c278e0457b9cc070ac0d662b5d' move scheduled
I20240502 01:59:28.503996 8961 rebalancer_tool.cc:1074] tablet b69d6e907b3b400f93721981732c006b: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '954cd49326bf405abd81182d75dc783c' move scheduled
I20240502 01:59:28.515682 8961 rebalancer_tool.cc:1074] tablet 7345f51bd6f84b3880838c2460bafb75: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:28.587285 8961 rebalancer_tool.cc:1074] tablet 1c54b912f802461c8cac092ac3222c87: '85e78945910a4e1abfa101d2edc3644e' -> '070558c278e0457b9cc070ac0d662b5d' move scheduled
I20240502 01:59:28.627107 8961 rebalancer_tool.cc:1084] tablet dcc5daef41bc44c4855f2bcb44e1c6d5: '86bc8001c3464b929cd5c873861baa0c' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move ignored: Illegal state: tablet dcc5daef41bc44c4855f2bcb44e1c6d5: replica 4e6e76b381af4cdc90bd3e9e6e8b5cf2 is already present
I20240502 01:59:28.627190 8961 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:29.137813 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:29.137928 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:29.138000 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:29.138088 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:29.138139 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:29.138188 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:29.138255 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:29.138310 8961 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
I20240502 01:59:29.302747 8961 rebalancer_tool.cc:1074] tablet 9f6c07e3923d47f29ef32561d7f438e1: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:29.387665 8961 rebalancer_tool.cc:1074] tablet 9313079c48c94ab1b7e022a91befafcd: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:29.401835 8961 rebalancer_tool.cc:1074] tablet 47a276e9780e448a9b3c6eab67925514: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:31.255529 8961 rebalancer_tool.cc:1136] tablet b69d6e907b3b400f93721981732c006b: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '954cd49326bf405abd81182d75dc783c' move completed: OK
I20240502 01:59:31.275535 8961 rebalancer_tool.cc:1136] tablet 9ecbf1c8ee8c46b195d2374a5db81ac0: '86bc8001c3464b929cd5c873861baa0c' -> '954cd49326bf405abd81182d75dc783c' move completed: OK
I20240502 01:59:32.486608 8961 rebalancer_tool.cc:1136] tablet ed7b67af7b1c4255a7c2377570b58e70: 'c6101231591a458384f4ea6d1f0e0ae5' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move completed: OK
I20240502 01:59:33.317261 8961 rebalancer_tool.cc:1136] tablet 39d9e48f67f34ebe850fc388ecd94ca3: '85e78945910a4e1abfa101d2edc3644e' -> '070558c278e0457b9cc070ac0d662b5d' move completed: OK
I20240502 01:59:33.382740 8961 rebalancer_tool.cc:1136] tablet 48eeb721a60c4be888041bd31cfeab7a: '86bc8001c3464b929cd5c873861baa0c' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move completed: OK
I20240502 01:59:33.397250 8961 rebalancer_tool.cc:1136] tablet 47a276e9780e448a9b3c6eab67925514: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:33.899498 8961 rebalancer_tool.cc:1123] tablet 8529a93abd6e4d62bbc0bddcf2fda78a: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move is abandoned: Not found: Server with UUID e7bcff2fef614e7a9981554c9e6db3c8 not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:34.557878 8961 rebalancer_tool.cc:1074] tablet 4e44267bd266407f9b72c66e3d3579bb: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:34.566936 8961 rebalancer_tool.cc:1136] tablet 9313079c48c94ab1b7e022a91befafcd: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:34.577853 8961 rebalancer_tool.cc:1136] tablet 9f6c07e3923d47f29ef32561d7f438e1: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:34.588963 8961 rebalancer_tool.cc:1136] tablet 7345f51bd6f84b3880838c2460bafb75: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:34.602484 8961 rebalancer_tool.cc:1136] tablet 1c54b912f802461c8cac092ac3222c87: '85e78945910a4e1abfa101d2edc3644e' -> '070558c278e0457b9cc070ac0d662b5d' move completed: OK
I20240502 01:59:34.610386 8961 rebalancer_tool.cc:1136] tablet 14e3c78c65334b52aca79d67c4bf45a2: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4f53d80e39be410d92d4766692aff1c6' move completed: OK
I20240502 01:59:34.618111 8961 rebalancer_tool.cc:1136] tablet 7fa358140eb742b587a4d6a203738b4f: 'c6101231591a458384f4ea6d1f0e0ae5' -> '070558c278e0457b9cc070ac0d662b5d' move completed: OK
I20240502 01:59:34.626590 8961 rebalancer_tool.cc:1136] tablet 4e44267bd266407f9b72c66e3d3579bb: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:35.250085 8961 rebalancer_tool.cc:1074] tablet 0359dfbfd11746f1bf62a465f14634ec: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:35.270181 8961 rebalancer_tool.cc:1074] tablet 17235e019e644b258266636fb6be6648: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:35.305718 8961 rebalancer_tool.cc:1074] tablet 4154be3f1f3c428580463c419d69892e: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:35.327495 8961 rebalancer_tool.cc:1074] tablet e0e05f96dbc542eb999c16dcc0620666: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:35.345017 8961 rebalancer_tool.cc:1136] tablet e0e05f96dbc542eb999c16dcc0620666: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:35.364570 8961 rebalancer_tool.cc:1136] tablet 4154be3f1f3c428580463c419d69892e: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:35.373204 8961 rebalancer_tool.cc:1136] tablet 17235e019e644b258266636fb6be6648: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:35.387449 8961 rebalancer_tool.cc:1136] tablet 0359dfbfd11746f1bf62a465f14634ec: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:35.395560 8961 rebalancer_tool.cc:1136] tablet 91db5c1b3d2f4d22b5882a61ea0395c7: '85e78945910a4e1abfa101d2edc3644e' -> '4f53d80e39be410d92d4766692aff1c6' move completed: OK
I20240502 01:59:35.418303 8961 rebalancer_tool.cc:1074] tablet e6ac26fa6de44f7ca68e6c0b6b959db7: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:35.432433 8961 rebalancer_tool.cc:1074] tablet 034cf1c3d47a4f7785c16d16815c229d: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:35.449807 8961 rebalancer_tool.cc:1074] tablet a4fb9af11abc49d496b5c3d8048c22c8: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:35.481954 8961 rebalancer_tool.cc:1136] tablet a4fb9af11abc49d496b5c3d8048c22c8: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:37.011746 8961 rebalancer_tool.cc:1136] tablet 034cf1c3d47a4f7785c16d16815c229d: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:37.706686 8961 rebalancer_tool.cc:1136] tablet e6ac26fa6de44f7ca68e6c0b6b959db7: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:38.311252 8961 rebalancer_tool.cc:1074] tablet d4d429fe60e74108bc0c9ca8509365f2: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '86bc8001c3464b929cd5c873861baa0c' move scheduled
I20240502 01:59:38.404830 8961 rebalancer_tool.cc:1074] tablet 47a276e9780e448a9b3c6eab67925514: '4e6e76b381af4cdc90bd3e9e6e8b5cf2' -> '954cd49326bf405abd81182d75dc783c' move scheduled
I20240502 01:59:38.571524 8961 rebalancer_tool.cc:1074] tablet 9f6c07e3923d47f29ef32561d7f438e1: '85e78945910a4e1abfa101d2edc3644e' -> '4f53d80e39be410d92d4766692aff1c6' move scheduled
I20240502 01:59:38.600133 8961 rebalancer_tool.cc:1074] tablet 17235e019e644b258266636fb6be6648: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:38.699146 8961 rebalancer_tool.cc:1074] tablet 9313079c48c94ab1b7e022a91befafcd: '85e78945910a4e1abfa101d2edc3644e' -> '86bc8001c3464b929cd5c873861baa0c' move scheduled
I20240502 01:59:41.224089 8961 rebalancer_tool.cc:1136] tablet 47a276e9780e448a9b3c6eab67925514: '4e6e76b381af4cdc90bd3e9e6e8b5cf2' -> '954cd49326bf405abd81182d75dc783c' move completed: OK
I20240502 01:59:43.924713 8961 rebalancer_tool.cc:1136] tablet d4d429fe60e74108bc0c9ca8509365f2: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '86bc8001c3464b929cd5c873861baa0c' move completed: OK
I20240502 01:59:43.958563 8961 rebalancer_tool.cc:1136] tablet 9313079c48c94ab1b7e022a91befafcd: '85e78945910a4e1abfa101d2edc3644e' -> '86bc8001c3464b929cd5c873861baa0c' move completed: OK
I20240502 01:59:44.028475 8961 rebalancer_tool.cc:1136] tablet 17235e019e644b258266636fb6be6648: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:44.120216 8961 rebalancer_tool.cc:1136] tablet 9f6c07e3923d47f29ef32561d7f438e1: '85e78945910a4e1abfa101d2edc3644e' -> '4f53d80e39be410d92d4766692aff1c6' move completed: OK
Illegal state: tablet server 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): unacceptable health status UNAVAILABLE
I20240502 01:59:44.611879 19501 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:44.616312 6642 catalog_manager.cc:5484] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 6 to 8, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New cstate: current_term: 2 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:44.618836 8508 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:44.625697 8904 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:44.644381 6894 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c55b199a24432e902cce9121fc0780 with delete_type TABLET_DATA_TOMBSTONED (TS e7bcff2fef614e7a9981554c9e6db3c8 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59160
I20240502 01:59:44.648638 19853 tablet_replica.cc:330] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8: stopping tablet replica
I20240502 01:59:44.648984 19853 raft_consensus.cc:2229] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:44.649149 19853 raft_consensus.cc:2258] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:44.649783 19853 ts_tablet_manager.cc:1875] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:44.653105 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.657341 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:44.662273 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.671967 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:44.675539 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:44.677299 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:44.686949 19853 ts_tablet_manager.cc:1888] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 01:59:44.691054 19853 log.cc:1196] T 44c55b199a24432e902cce9121fc0780 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/44c55b199a24432e902cce9121fc0780
I20240502 01:59:44.691758 6630 catalog_manager.cc:4830] TS e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): tablet 44c55b199a24432e902cce9121fc0780 (table rebalance_test_table_4 [id=f2b695d4b3e54fdfad81f019e9ab7b4a]) successfully deleted
I20240502 01:59:44.700798 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:44.738484 8641 consensus_queue.cc:237] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:44.739957 8508 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 01:59:44.740710 8902 raft_consensus.cc:1261] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d [term 3 FOLLOWER]: Refusing update from remote peer 4f53d80e39be410d92d4766692aff1c6: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 01:59:44.740723 19319 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:59:44.741899 19319 consensus_queue.cc:1035] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:59:44.744237 19319 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P 4f53d80e39be410d92d4766692aff1c6 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:44.746124 8902 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:44.749684 6636 catalog_manager.cc:5484] T 034cf1c3d47a4f7785c16d16815c229d P 070558c278e0457b9cc070ac0d662b5d reported cstate change: config changed from index 9 to 11, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New cstate: current_term: 3 leader_uuid: "4f53d80e39be410d92d4766692aff1c6" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:44.752511 6763 tablet_service.cc:1514] Processing DeleteTablet for tablet 034cf1c3d47a4f7785c16d16815c229d with delete_type TABLET_DATA_TOMBSTONED (TS 85e78945910a4e1abfa101d2edc3644e not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55860
I20240502 01:59:44.759372 19890 tablet_replica.cc:330] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e: stopping tablet replica
I20240502 01:59:44.759639 19890 raft_consensus.cc:2229] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:59:44.759824 8508 raft_consensus.cc:2932] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 01:59:44.762380 19890 raft_consensus.cc:2258] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:59:44.763063 19890 ts_tablet_manager.cc:1875] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:59:44.767822 19890 ts_tablet_manager.cc:1888] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240502 01:59:44.767975 19890 log.cc:1196] T 034cf1c3d47a4f7785c16d16815c229d P 85e78945910a4e1abfa101d2edc3644e: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/034cf1c3d47a4f7785c16d16815c229d
I20240502 01:59:44.768518 6629 catalog_manager.cc:4830] TS 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): tablet 034cf1c3d47a4f7785c16d16815c229d (table rebalance_test_table_6 [id=331b273e69914efb86a2391b30c6d6cf]) successfully deleted
I20240502 01:59:44.792094 8956 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task43FDSt/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 01:59:24.579299 8960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:24.581539 8960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:24.588061 8960 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:59:25.682126 8960 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:59:26.205125 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.205382 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.205437 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.205483 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.205531 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.205605 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.205683 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.205772 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.205834 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.205893 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.205927 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.205965 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.206012 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.206045 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.206077 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:26.283877 8960 rebalancer_tool.cc:1074] tablet 4154be3f1f3c428580463c419d69892e: '85e78945910a4e1abfa101d2edc3644e' -> '070558c278e0457b9cc070ac0d662b5d' move scheduled
I20240502 01:59:26.374425 8960 rebalancer_tool.cc:1074] tablet 9313079c48c94ab1b7e022a91befafcd: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '954cd49326bf405abd81182d75dc783c' move scheduled
I20240502 01:59:26.437129 8960 rebalancer_tool.cc:1084] tablet 17235e019e644b258266636fb6be6648: '86bc8001c3464b929cd5c873861baa0c' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }.
Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:26.437253 8960 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:26.919559 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.919660 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.919737 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.919797 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.919832 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.919867 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.919901 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.919932 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.919978 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.920037 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.920104 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 070558c278e0457b9cc070ac0d662b5d to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.920161 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.920212 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.920243 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.920269 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.920300 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.920336 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.920369 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.920418 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.920454 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.920487 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.920511 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:27.239976 8960 rebalancer_tool.cc:1074] tablet a4fb9af11abc49d496b5c3d8048c22c8: '86bc8001c3464b929cd5c873861baa0c' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move scheduled
I20240502 01:59:27.585225 8960 rebalancer_tool.cc:1074] tablet a66b932d884e4df8a78841103d88ded0: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4f53d80e39be410d92d4766692aff1c6' move scheduled
I20240502 01:59:27.665535 8960 rebalancer_tool.cc:1074] tablet e6ac26fa6de44f7ca68e6c0b6b959db7: 'c6101231591a458384f4ea6d1f0e0ae5' -> '954cd49326bf405abd81182d75dc783c' move scheduled
I20240502 01:59:27.739032 8960 rebalancer_tool.cc:1074] tablet 44c55b199a24432e902cce9121fc0780: '85e78945910a4e1abfa101d2edc3644e' -> '070558c278e0457b9cc070ac0d662b5d' move scheduled
I20240502 01:59:28.608799 8960 rebalancer_tool.cc:1074] tablet 48eeb721a60c4be888041bd31cfeab7a: '86bc8001c3464b929cd5c873861baa0c' -> '4f53d80e39be410d92d4766692aff1c6' move scheduled
I20240502 01:59:28.640460 8960 rebalancer_tool.cc:1084] tablet e0e05f96dbc542eb999c16dcc0620666: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move ignored: Illegal state: tablet e0e05f96dbc542eb999c16dcc0620666: replica 4e6e76b381af4cdc90bd3e9e6e8b5cf2 is already present
I20240502 01:59:28.640553 8960 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:29.066423 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:29.066581 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:29.066641 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:29.066684 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:29.066722 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:29.066757 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:29.066795 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:29.066862 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:29.066905 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
I20240502 01:59:29.117133 8960 rebalancer_tool.cc:1074] tablet ed7b67af7b1c4255a7c2377570b58e70: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:29.201030 8960 rebalancer_tool.cc:1074] tablet e0e05f96dbc542eb999c16dcc0620666: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:29.298239 8960 rebalancer_tool.cc:1074] tablet 47a276e9780e448a9b3c6eab67925514: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:29.361064 8960 rebalancer_tool.cc:1074] tablet 9ecbf1c8ee8c46b195d2374a5db81ac0: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:29.405880 8960 rebalancer_tool.cc:1074] tablet c8c237c50773494abf8813494be0f184: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:32.074445 8960 rebalancer_tool.cc:1136] tablet ed7b67af7b1c4255a7c2377570b58e70: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:32.699115 8960 rebalancer_tool.cc:1136] tablet 9313079c48c94ab1b7e022a91befafcd: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '954cd49326bf405abd81182d75dc783c' move completed: OK
I20240502 01:59:32.707564 8960 rebalancer_tool.cc:1136] tablet 4154be3f1f3c428580463c419d69892e: '85e78945910a4e1abfa101d2edc3644e' -> '070558c278e0457b9cc070ac0d662b5d' move completed: OK
I20240502 01:59:32.777791 8960 rebalancer_tool.cc:1136] tablet a4fb9af11abc49d496b5c3d8048c22c8: '86bc8001c3464b929cd5c873861baa0c' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move completed: OK
I20240502 01:59:34.493544 8960 rebalancer_tool.cc:1074] tablet 9f6c07e3923d47f29ef32561d7f438e1: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:34.549844 8960 rebalancer_tool.cc:1136] tablet 9f6c07e3923d47f29ef32561d7f438e1: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:34.562834 8960 rebalancer_tool.cc:1136] tablet 47a276e9780e448a9b3c6eab67925514: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:34.572029 8960 rebalancer_tool.cc:1136] tablet 9ecbf1c8ee8c46b195d2374a5db81ac0: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:34.585866 8960 rebalancer_tool.cc:1136] tablet e0e05f96dbc542eb999c16dcc0620666: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:34.600930 8960 rebalancer_tool.cc:1136] tablet c8c237c50773494abf8813494be0f184: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:34.611619 8960 rebalancer_tool.cc:1136] tablet 48eeb721a60c4be888041bd31cfeab7a: '86bc8001c3464b929cd5c873861baa0c' -> '4f53d80e39be410d92d4766692aff1c6' move completed: OK
I20240502 01:59:34.621275 8960 rebalancer_tool.cc:1136] tablet a66b932d884e4df8a78841103d88ded0: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4f53d80e39be410d92d4766692aff1c6' move completed: OK
I20240502 01:59:34.629933 8960 rebalancer_tool.cc:1136] tablet e6ac26fa6de44f7ca68e6c0b6b959db7: 'c6101231591a458384f4ea6d1f0e0ae5' -> '954cd49326bf405abd81182d75dc783c' move completed: OK
I20240502 01:59:34.657605 8960 rebalancer_tool.cc:1136] tablet 44c55b199a24432e902cce9121fc0780: '85e78945910a4e1abfa101d2edc3644e' -> '070558c278e0457b9cc070ac0d662b5d' move completed: OK
I20240502 01:59:34.672583 8960 rebalancer_tool.cc:1074] tablet 91db5c1b3d2f4d22b5882a61ea0395c7: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:34.720315 8960 rebalancer_tool.cc:1074] tablet 14e3c78c65334b52aca79d67c4bf45a2: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:34.729436 8960 rebalancer_tool.cc:1074] tablet 7345f51bd6f84b3880838c2460bafb75: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:34.762293 8960 rebalancer_tool.cc:1074] tablet 39d9e48f67f34ebe850fc388ecd94ca3: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:34.795264 8960 rebalancer_tool.cc:1074] tablet 8529a93abd6e4d62bbc0bddcf2fda78a: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:35.501791 8960 rebalancer_tool.cc:1136] tablet 7345f51bd6f84b3880838c2460bafb75: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:35.924700 8960 rebalancer_tool.cc:1136] tablet 8529a93abd6e4d62bbc0bddcf2fda78a: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:36.965835 8960 rebalancer_tool.cc:1074] tablet 347b161e9ec84640a0668836ce9e29eb: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:37.021514 8960 rebalancer_tool.cc:1074] tablet a0548274c47148d9b611fd3c978a0c34: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:37.050896 8960 rebalancer_tool.cc:1136] tablet a0548274c47148d9b611fd3c978a0c34: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:37.079993 8960 rebalancer_tool.cc:1136] tablet 347b161e9ec84640a0668836ce9e29eb: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:37.103732 8960 rebalancer_tool.cc:1136] tablet 39d9e48f67f34ebe850fc388ecd94ca3: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:37.947659 8960 rebalancer_tool.cc:1136] tablet 91db5c1b3d2f4d22b5882a61ea0395c7: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:38.002161 8960 rebalancer_tool.cc:1074] tablet 17235e019e644b258266636fb6be6648: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:38.131125 8960 rebalancer_tool.cc:1136] tablet 17235e019e644b258266636fb6be6648: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:38.172621 8960 rebalancer_tool.cc:1136] tablet 14e3c78c65334b52aca79d67c4bf45a2: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
W20240502 01:59:38.571691 8960 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:38.602237 8960 rebalancer_tool.cc:1074] tablet c8c237c50773494abf8813494be0f184: '4e6e76b381af4cdc90bd3e9e6e8b5cf2' -> '' move scheduled
I20240502 01:59:38.622368 8960 rebalancer_tool.cc:1084] tablet 9f6c07e3923d47f29ef32561d7f438e1: '85e78945910a4e1abfa101d2edc3644e' -> '4f53d80e39be410d92d4766692aff1c6' move ignored: Illegal state: tablet 9f6c07e3923d47f29ef32561d7f438e1: replica 4f53d80e39be410d92d4766692aff1c6 is already present
I20240502 01:59:38.622525 8960 rebalancer_tool.cc:756] re-synchronizing cluster state
I20240502 01:59:39.053090 8960 rebalancer_tool.cc:1074] tablet dcc5daef41bc44c4855f2bcb44e1c6d5: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:40.696668 8960 rebalancer_tool.cc:1136] tablet dcc5daef41bc44c4855f2bcb44e1c6d5: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:40.738199 8960 rebalancer_tool.cc:1136] tablet c8c237c50773494abf8813494be0f184: '4e6e76b381af4cdc90bd3e9e6e8b5cf2' -> '' move completed: OK
I20240502 01:59:41.197762 8960 rebalancer_tool.cc:1074] tablet 9ecbf1c8ee8c46b195d2374a5db81ac0: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:42.191504 8960 rebalancer_tool.cc:1136] tablet 9ecbf1c8ee8c46b195d2374a5db81ac0: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:42.681351 8960 rebalancer_tool.cc:1074] tablet 1c54b912f802461c8cac092ac3222c87: '070558c278e0457b9cc070ac0d662b5d' -> '' move scheduled
I20240502 01:59:42.950250 8960 rebalancer_tool.cc:1074] tablet 0359dfbfd11746f1bf62a465f14634ec: '954cd49326bf405abd81182d75dc783c' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move scheduled
W20240502 01:59:44.187952 19229 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:59:44.181906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.111.199:41741 (local address 127.0.0.1:36362)
0502 01:59:44.182053 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0502 01:59:44.182070 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0502 01:59:44.182193 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:59:44.187348 (+ 5155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":49}
I20240502 01:59:44.298152 8960 rebalancer_tool.cc:1123] tablet 1c54b912f802461c8cac092ac3222c87: '070558c278e0457b9cc070ac0d662b5d' -> '' move is abandoned: Not found: Server with UUID 070558c278e0457b9cc070ac0d662b5d not a member of the config. RaftConfig: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 01:59:44.298255 8960 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): unacceptable health status UNAVAILABLE
I20240502 01:59:44.828820 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.854735 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:44.866565 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:59:44.871730 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:44.877573 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:44.878991 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:44.886902 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:44.905073 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.985110 8957 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task43FDSt/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 01:59:24.583369 8962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:24.583626 8962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:24.587863 8962 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:59:25.700273 8962 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:59:26.258859 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.259069 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.260049 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.260143 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.260200 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.260257 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.260306 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.260391 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.261350 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.261430 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.261515 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.261576 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.261647 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.261757 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 954cd49326bf405abd81182d75dc783c
I20240502 01:59:26.388021 8962 rebalancer_tool.cc:1074] tablet a0548274c47148d9b611fd3c978a0c34: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move scheduled
I20240502 01:59:26.444906 8962 rebalancer_tool.cc:1074] tablet 7345f51bd6f84b3880838c2460bafb75: '86bc8001c3464b929cd5c873861baa0c' -> '954cd49326bf405abd81182d75dc783c' move scheduled
I20240502 01:59:26.514496 8962 rebalancer_tool.cc:1074] tablet 347b161e9ec84640a0668836ce9e29eb: '85e78945910a4e1abfa101d2edc3644e' -> '070558c278e0457b9cc070ac0d662b5d' move scheduled
I20240502 01:59:26.632323 8962 rebalancer_tool.cc:1074] tablet 818dbc9a8e054eeaa122e7919a311806: 'c6101231591a458384f4ea6d1f0e0ae5' -> '4f53d80e39be410d92d4766692aff1c6' move scheduled
I20240502 01:59:26.822791 8962 rebalancer_tool.cc:1074] tablet 8529a93abd6e4d62bbc0bddcf2fda78a: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move scheduled
I20240502 01:59:26.834851 8962 rebalancer_tool.cc:1084] tablet 17235e019e644b258266636fb6be6648: '86bc8001c3464b929cd5c873861baa0c' -> '954cd49326bf405abd81182d75dc783c' move ignored: Illegal state: tablet 17235e019e644b258266636fb6be6648: replica 954cd49326bf405abd81182d75dc783c is already present
I20240502 01:59:26.834966 8962 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:27.570503 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.570636 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.570725 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.570783 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.570817 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.570892 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.570948 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.570976 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.571015 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.571053 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.571086 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.571138 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.571815 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.571926 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.571974 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.572014 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.572060 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.572103 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.572160 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:27.599306 8962 rebalancer_tool.cc:1074] tablet ed7b67af7b1c4255a7c2377570b58e70: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:27.623548 8962 rebalancer_tool.cc:1084] tablet 47a276e9780e448a9b3c6eab67925514: '85e78945910a4e1abfa101d2edc3644e' -> '4f53d80e39be410d92d4766692aff1c6' move ignored: Illegal state: tablet 47a276e9780e448a9b3c6eab67925514: replica 4f53d80e39be410d92d4766692aff1c6 is already present
I20240502 01:59:27.623672 8962 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:27.998070 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.998277 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.998368 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.998442 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.998500 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.998565 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.998615 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.998658 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.998697 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.998811 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.998867 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.998907 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.998957 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.999013 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.999058 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:28.007936 8962 rebalancer_tool.cc:1074] tablet 17235e019e644b258266636fb6be6648: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:28.079586 8962 rebalancer_tool.cc:1074] tablet 1c54b912f802461c8cac092ac3222c87: '85e78945910a4e1abfa101d2edc3644e' -> '954cd49326bf405abd81182d75dc783c' move scheduled
I20240502 01:59:28.356954 8962 rebalancer_tool.cc:1084] tablet 7fa358140eb742b587a4d6a203738b4f: 'c6101231591a458384f4ea6d1f0e0ae5' -> '4f53d80e39be410d92d4766692aff1c6' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 01:59:28.357036 8962 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:28.695585 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.695670 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.695704 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.695832 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.695914 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.696008 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:28.696049 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.696106 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:28.696152 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:28.696192 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.696242 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:28.696282 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:28.696333 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.696421 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.696481 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.696527 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.696557 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:28.711988 8962 rebalancer_tool.cc:1074] tablet 9313079c48c94ab1b7e022a91befafcd: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:28.789681 8962 rebalancer_tool.cc:1074] tablet d4d429fe60e74108bc0c9ca8509365f2: '86bc8001c3464b929cd5c873861baa0c' -> '954cd49326bf405abd81182d75dc783c' move scheduled
I20240502 01:59:28.875912 8962 rebalancer_tool.cc:1074] tablet 9f6c07e3923d47f29ef32561d7f438e1: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:28.945837 8962 rebalancer_tool.cc:1074] tablet 9ecbf1c8ee8c46b195d2374a5db81ac0: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:31.919126 8962 rebalancer_tool.cc:1136] tablet 9f6c07e3923d47f29ef32561d7f438e1: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:31.953508 8962 rebalancer_tool.cc:1136] tablet 7345f51bd6f84b3880838c2460bafb75: '86bc8001c3464b929cd5c873861baa0c' -> '954cd49326bf405abd81182d75dc783c' move completed: OK
I20240502 01:59:33.483096 8962 rebalancer_tool.cc:1136] tablet 347b161e9ec84640a0668836ce9e29eb: '85e78945910a4e1abfa101d2edc3644e' -> '070558c278e0457b9cc070ac0d662b5d' move completed: OK
I20240502 01:59:33.498942 8962 rebalancer_tool.cc:1136] tablet 1c54b912f802461c8cac092ac3222c87: '85e78945910a4e1abfa101d2edc3644e' -> '954cd49326bf405abd81182d75dc783c' move completed: OK
I20240502 01:59:33.559648 8962 rebalancer_tool.cc:1136] tablet ed7b67af7b1c4255a7c2377570b58e70: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:33.691287 8962 rebalancer_tool.cc:1074] tablet 4154be3f1f3c428580463c419d69892e: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:33.780351 8962 rebalancer_tool.cc:1074] tablet 39d9e48f67f34ebe850fc388ecd94ca3: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:34.256083 8962 rebalancer_tool.cc:1136] tablet 4154be3f1f3c428580463c419d69892e: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:34.281991 8962 rebalancer_tool.cc:1136] tablet 9ecbf1c8ee8c46b195d2374a5db81ac0: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:34.309878 8962 rebalancer_tool.cc:1136] tablet 9313079c48c94ab1b7e022a91befafcd: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:34.323419 8962 rebalancer_tool.cc:1136] tablet 17235e019e644b258266636fb6be6648: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:34.371660 8962 rebalancer_tool.cc:1136] tablet d4d429fe60e74108bc0c9ca8509365f2: '86bc8001c3464b929cd5c873861baa0c' -> '954cd49326bf405abd81182d75dc783c' move completed: OK
I20240502 01:59:34.379700 8962 rebalancer_tool.cc:1136] tablet a0548274c47148d9b611fd3c978a0c34: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move completed: OK
I20240502 01:59:34.425813 8962 rebalancer_tool.cc:1136] tablet 818dbc9a8e054eeaa122e7919a311806: 'c6101231591a458384f4ea6d1f0e0ae5' -> '4f53d80e39be410d92d4766692aff1c6' move completed: OK
I20240502 01:59:34.513463 8962 rebalancer_tool.cc:1136] tablet 8529a93abd6e4d62bbc0bddcf2fda78a: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move completed: OK
I20240502 01:59:34.536962 8962 rebalancer_tool.cc:1074] tablet 034cf1c3d47a4f7785c16d16815c229d: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:34.548403 8962 rebalancer_tool.cc:1074] tablet 14e3c78c65334b52aca79d67c4bf45a2: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:35.092571 8962 rebalancer_tool.cc:1136] tablet 034cf1c3d47a4f7785c16d16815c229d: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:35.603683 8962 rebalancer_tool.cc:1123] tablet 39d9e48f67f34ebe850fc388ecd94ca3: '86bc8001c3464b929cd5c873861baa0c' -> '' move is abandoned: Not found: Server with UUID 86bc8001c3464b929cd5c873861baa0c not a member of the config. RaftConfig: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:35.628453 8962 rebalancer_tool.cc:1136] tablet 14e3c78c65334b52aca79d67c4bf45a2: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:36.354493 8962 rebalancer_tool.cc:1074] tablet a4fb9af11abc49d496b5c3d8048c22c8: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4f53d80e39be410d92d4766692aff1c6' move scheduled
I20240502 01:59:36.533854 8962 rebalancer_tool.cc:1074] tablet dcc5daef41bc44c4855f2bcb44e1c6d5: 'c6101231591a458384f4ea6d1f0e0ae5' -> '070558c278e0457b9cc070ac0d662b5d' move scheduled
I20240502 01:59:36.715762 8962 rebalancer_tool.cc:1074] tablet c8c237c50773494abf8813494be0f184: '86bc8001c3464b929cd5c873861baa0c' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move scheduled
I20240502 01:59:36.976217 8962 rebalancer_tool.cc:1084] tablet a66b932d884e4df8a78841103d88ded0: '85e78945910a4e1abfa101d2edc3644e' -> '954cd49326bf405abd81182d75dc783c' move ignored: Illegal state: Leader has not yet committed an operation in its own term
I20240502 01:59:36.976310 8962 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:37.316522 8962 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 4e6e76b381af4cdc90bd3e9e6e8b5cf2 to server 86bc8001c3464b929cd5c873861baa0c
I20240502 01:59:37.535827 8962 rebalancer_tool.cc:1074] tablet 44c55b199a24432e902cce9121fc0780: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '954cd49326bf405abd81182d75dc783c' move scheduled
I20240502 01:59:37.569298 8962 rebalancer_tool.cc:1074] tablet 48eeb721a60c4be888041bd31cfeab7a: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:37.740384 8962 rebalancer_tool.cc:1084] tablet 91db5c1b3d2f4d22b5882a61ea0395c7: '86bc8001c3464b929cd5c873861baa0c' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move ignored: Illegal state: Request specified cas_config_opid_index of 8 but the committed config has opid_index of 9
I20240502 01:59:37.740491 8962 rebalancer_tool.cc:756] re-synchronizing cluster state
I20240502 01:59:38.325525 8962 rebalancer_tool.cc:1074] tablet 47a276e9780e448a9b3c6eab67925514: '4e6e76b381af4cdc90bd3e9e6e8b5cf2' -> '86bc8001c3464b929cd5c873861baa0c' move scheduled
I20240502 01:59:38.446486 8962 rebalancer_tool.cc:1074] tablet e6ac26fa6de44f7ca68e6c0b6b959db7: '85e78945910a4e1abfa101d2edc3644e' -> '4f53d80e39be410d92d4766692aff1c6' move scheduled
I20240502 01:59:38.590425 8962 rebalancer_tool.cc:1074] tablet 0359dfbfd11746f1bf62a465f14634ec: '070558c278e0457b9cc070ac0d662b5d' -> '954cd49326bf405abd81182d75dc783c' move scheduled
I20240502 01:59:38.616357 8962 rebalancer_tool.cc:1084] tablet d4d429fe60e74108bc0c9ca8509365f2: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '86bc8001c3464b929cd5c873861baa0c' move ignored: Illegal state: tablet d4d429fe60e74108bc0c9ca8509365f2: replica 86bc8001c3464b929cd5c873861baa0c is already present
I20240502 01:59:38.616451 8962 rebalancer_tool.cc:756] re-synchronizing cluster state
I20240502 01:59:39.159492 8962 rebalancer_tool.cc:1074] tablet 9ecbf1c8ee8c46b195d2374a5db81ac0: '954cd49326bf405abd81182d75dc783c' -> '85e78945910a4e1abfa101d2edc3644e' move scheduled
I20240502 01:59:43.228952 8962 rebalancer_tool.cc:1136] tablet c8c237c50773494abf8813494be0f184: '86bc8001c3464b929cd5c873861baa0c' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move completed: Incomplete: tablet c8c237c50773494abf8813494be0f184, TS 86bc8001c3464b929cd5c873861baa0c -> TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 move failed, target replica disappeared
I20240502 01:59:43.316860 8962 rebalancer_tool.cc:1136] tablet dcc5daef41bc44c4855f2bcb44e1c6d5: 'c6101231591a458384f4ea6d1f0e0ae5' -> '070558c278e0457b9cc070ac0d662b5d' move completed: OK
I20240502 01:59:43.
I20240502 01:59:44.989696 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:44.995296 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:44.999907 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:45.002207 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:45.004048 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:45.004494 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:45.015477 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:45.030555 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:45.035681 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:45.038844 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:45.041884 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:45.042297 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:45.044987 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:45.054118 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:45.134203 8955 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task43FDSt/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 01:59:24.543033 8959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:24.543392 8959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:24.549469 8959 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:59:25.703977 8959 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:59:26.253175 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.253391 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.253439 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.253532 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.253589 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.253635 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.253679 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.253762 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.253846 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.253917 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.253965 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.254015 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.254065 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.254114 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
I20240502 01:59:26.360963 8959 rebalancer_tool.cc:1074] tablet 034cf1c3d47a4f7785c16d16815c229d: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4f53d80e39be410d92d4766692aff1c6' move scheduled
I20240502 01:59:26.444614 8959 rebalancer_tool.cc:1074] tablet 17235e019e644b258266636fb6be6648: '86bc8001c3464b929cd5c873861baa0c' -> '954cd49326bf405abd81182d75dc783c' move scheduled
I20240502 01:59:26.533685 8959 rebalancer_tool.cc:1074] tablet 7345f51bd6f84b3880838c2460bafb75: '85e78945910a4e1abfa101d2edc3644e' -> '070558c278e0457b9cc070ac0d662b5d' move scheduled
I20240502 01:59:26.613821 8959 rebalancer_tool.cc:1084] tablet 818dbc9a8e054eeaa122e7919a311806: 'c6101231591a458384f4ea6d1f0e0ae5' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 01:59:26.613929 8959 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:27.045686 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.045917 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.046097 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.046168 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.046236 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.046308 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.046365 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.046447 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.046607 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.046684 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.046741 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.046819 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.046877 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.046924 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.046974 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.047022 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.047073 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.047156 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.047209 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.047255 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:27.215075 8959 rebalancer_tool.cc:1074] tablet e6ac26fa6de44f7ca68e6c0b6b959db7: 'c6101231591a458384f4ea6d1f0e0ae5' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move scheduled
I20240502 01:59:27.475144 8959 rebalancer_tool.cc:1074] tablet 47a276e9780e448a9b3c6eab67925514: '85e78945910a4e1abfa101d2edc3644e' -> '4f53d80e39be410d92d4766692aff1c6' move scheduled
I20240502 01:59:27.597646 8959 rebalancer_tool.cc:1074] tablet 48eeb721a60c4be888041bd31cfeab7a: '86bc8001c3464b929cd5c873861baa0c' -> '954cd49326bf405abd81182d75dc783c' move scheduled
I20240502 01:59:27.689275 8959 rebalancer_tool.cc:1074] tablet e0e05f96dbc542eb999c16dcc0620666: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move scheduled
I20240502 01:59:27.773288 8959 rebalancer_tool.cc:1074] tablet 0359dfbfd11746f1bf62a465f14634ec: 'c6101231591a458384f4ea6d1f0e0ae5' -> '070558c278e0457b9cc070ac0d662b5d' move scheduled
I20240502 01:59:27.883394 8959 rebalancer_tool.cc:1084] tablet 4e44267bd266407f9b72c66e3d3579bb: '85e78945910a4e1abfa101d2edc3644e' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 01:59:27.883497 8959 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:28.460896 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.461030 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:28.461076 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.461102 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.461215 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:28.461249 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.461293 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:28.461323 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.461352 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:28.461385 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.461417 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:28.506031 8959 rebalancer_tool.cc:1074] tablet 8529a93abd6e4d62bbc0bddcf2fda78a: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:28.558529 8959 rebalancer_tool.cc:1074] tablet ed7b67af7b1c4255a7c2377570b58e70: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:28.705962 8959 rebalancer_tool.cc:1074] tablet b69d6e907b3b400f93721981732c006b: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move scheduled
I20240502 01:59:28.724823 8959 rebalancer_tool.cc:1074] tablet 39d9e48f67f34ebe850fc388ecd94ca3: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:28.974891 8959 rebalancer_tool.cc:1084] tablet d4d429fe60e74108bc0c9ca8509365f2: '86bc8001c3464b929cd5c873861baa0c' -> '4f53d80e39be410d92d4766692aff1c6' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 01:59:28.974999 8959 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:29.353271 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:29.353363 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:29.353572 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:29.353643 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:29.353688 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:29.353732 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:29.353812 8959 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 954cd49326bf405abd81182d75dc783c
I20240502 01:59:29.364177 8959 rebalancer_tool.cc:1074] tablet a4fb9af11abc49d496b5c3d8048c22c8: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:29.373502 8959 rebalancer_tool.cc:1074] tablet 347b161e9ec84640a0668836ce9e29eb: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:31.215991 8959 rebalancer_tool.cc:1136] tablet 347b161e9ec84640a0668836ce9e29eb: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:31.247262 8959 rebalancer_tool.cc:1136] tablet 0359dfbfd11746f1bf62a465f14634ec: 'c6101231591a458384f4ea6d1f0e0ae5' -> '070558c278e0457b9cc070ac0d662b5d' move completed: OK
I20240502 01:59:31.757905 8959 rebalancer_tool.cc:1123] tablet e0e05f96dbc542eb999c16dcc0620666: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move is abandoned: Not found: Server with UUID e7bcff2fef614e7a9981554c9e6db3c8 not a member of the config. RaftConfig: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4e6e76b381af4cdc90bd3e9e6e8b5cf2" member_type: VOTER last_known_addr { host: "127.27.111.199" port: 41741 } }
I20240502 01:59:31.799170 8959 rebalancer_tool.cc:1136] tablet 17235e019e644b258266636fb6be6648: '86bc8001c3464b929cd5c873861baa0c' -> '954cd49326bf405abd81182d75dc783c' move completed: OK
I20240502 01:59:33.890029 8959 rebalancer_tool.cc:1136] tablet 8529a93abd6e4d62bbc0bddcf2fda78a: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:33.901852 8959 rebalancer_tool.cc:1136] tablet 47a276e9780e448a9b3c6eab67925514: '85e78945910a4e1abfa101d2edc3644e' -> '4f53d80e39be410d92d4766692aff1c6' move completed: OK
I20240502 01:59:33.912587 8959 rebalancer_tool.cc:1136] tablet 48eeb721a60c4be888041bd31cfeab7a: '86bc8001c3464b929cd5c873861baa0c' -> '954cd49326bf405abd81182d75dc783c' move completed: OK
I20240502 01:59:33.924672 8959 rebalancer_tool.cc:1074] tablet 9ecbf1c8ee8c46b195d2374a5db81ac0: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:33.949811 8959 rebalancer_tool.cc:1074] tablet c8c237c50773494abf8813494be0f184: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:33.961462 8959 rebalancer_tool.cc:1136] tablet 9ecbf1c8ee8c46b195d2374a5db81ac0: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:33.969379 8959 rebalancer_tool.cc:1136] tablet a4fb9af11abc49d496b5c3d8048c22c8: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:34.130467 8959 rebalancer_tool.cc:1136] tablet b69d6e907b3b400f93721981732c006b: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move completed: OK
I20240502 01:59:34.207175 8959 rebalancer_tool.cc:1136] tablet c8c237c50773494abf8813494be0f184: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:34.287895 8959 rebalancer_tool.cc:1136] tablet ed7b67af7b1c4255a7c2377570b58e70: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:34.299952 8959 rebalancer_tool.cc:1136] tablet 39d9e48f67f34ebe850fc388ecd94ca3: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:34.315346 8959 rebalancer_tool.cc:1136] tablet 034cf1c3d47a4f7785c16d16815c229d: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4f53d80e39be410d92d4766692aff1c6' move completed: OK
I20240502 01:59:34.401793 8959 rebalancer_tool.cc:1136] tablet 7345f51bd6f84b3880838c2460bafb75: '85e78945910a4e1abfa101d2edc3644e' -> '070558c278e0457b9cc070ac0d662b5d' move completed: OK
I20240502 01:59:34.440660 8959 rebalancer_tool.cc:1136] tablet e6ac26fa6de44f7ca68e6c0b6b959db7: 'c6101231591a458384f4ea6d1f0e0ae5' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move completed: OK
I20240502 01:59:34.516719 8959 rebalancer_tool.cc:1074] tablet 91db5c1b3d2f4d22b5882a61ea0395c7: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:34.539670 8959 rebalancer_tool.cc:1074] tablet a66b932d884e4df8a78841103d88ded0: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:34.560199 8959 rebalancer_tool.cc:1074] tablet 4154be3f1f3c428580463c419d69892e: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:34.578564 8959 rebalancer_tool.cc:1074] tablet 14e3c78c65334b52aca79d67c4bf45a2: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:34.591753 8959 rebalancer_tool.cc:1074] tablet 818dbc9a8e054eeaa122e7919a311806: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:38.040244 8959 rebalancer_tool.cc:1136] tablet 818dbc9a8e054eeaa122e7919a311806: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:38.149206 8959 rebalancer_tool.cc:1136] tablet 14e3c78c65334b52aca79d67c4bf45a2: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:38.227463 8959 rebalancer_tool.cc:1136] tablet 4154be3f1f3c428580463c419d69892e: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:38.681643 8959 rebalancer_tool.cc:1136] tablet 91db5c1b3d2f4d22b5882a61ea0395c7: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:38.707245 8959 rebalancer_tool.cc:1074] tablet 9f6c07e3923d47f29ef32561d7f438e1: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:38.813155 8959 rebalancer_tool.cc:1074] tablet 44c55b199a24432e902cce9121fc0780: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:38.836122 8959 rebalancer_tool.cc:1074] tablet a0548274c47148d9b611fd3c978a0c34: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:40.231712 8959 rebalancer_tool.cc:1136] tablet 44c55b199a24432e902cce9121fc0780: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:40.261106 8959 rebalancer_tool.cc:1136] tablet 9f6c07e3923d47f29ef32561d7f438e1: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:40.338958 8959 rebalancer_tool.cc:1136] tablet a66b932d884e4df8a78841103d88ded0: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:41.146445 8959 rebalancer_tool.cc:1136] tablet a0548274c47148d9b611fd3c978a0c34: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:41.795075 8959 rebalancer_tool.cc:1074] tablet 1c54b912f802461c8cac092ac3222c87: '070558c278e0457b9cc070ac0d662b5d' -> '' move scheduled
I20240502 01:59:41.872267 8959 rebalancer_tool.cc:1074] tablet 034cf1c3d47a4f7785c16d16815c229d: '85e78945910a4e1abfa101d2edc3644e' -> '954cd49326bf405abd81182d75dc783c' move scheduled
I20240502 01:59:41.884538 8959 rebalancer_tool.cc:1074] tablet 47a276e9780e448a9b3c6eab67925514: '954cd49326bf405abd81182d75dc783c' -> '' move scheduled
I20240502 01:59:44.200968 8959 rebalancer_tool.cc:1136] tablet 47a276e9780e448a9b3c6eab67925514: '954cd49326bf405abd81182d75dc783c' -> '' move completed: OK
I20240502 01:59:44.784730 8959 rebalancer_tool.cc:1136] tablet 1c54b912f802461c8cac092ac3222c87: '070558c278e0457b9cc070ac0d662b5d' -> '' move completed: OK
I20240502 01:59:44.800118 8959 rebalancer_tool.cc:1136] tablet 034cf1c3d47a4f7785c16d16815c229d: '85e78945910a4e1abfa101d2edc3644e' -> '954cd49326bf405abd81182d75dc783c' move completed: OK
Illegal state: tablet server 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): unacceptable health status UNAVAILABLE
I20240502 01:59:45.376422 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:45.377111 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:45.378842 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:45.381342 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:45.382782 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:45.394953 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 01:59:45.399159 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:45.415586 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:45.702831 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:45.703264 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:45.704167 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:45.709416 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:45.710551 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:45.726075 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:45.737876 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:59:45.954910 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:46.034900 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:46.037248 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:46.039294 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:46.040604 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:46.051393 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:46.052203 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:46.065341 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 01:59:46.329514 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:59:46.376327 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:59:46.382474 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:46.385123 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:46.389552 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:46.389803 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:46.393401 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:59:46.399902 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:59:46.407231 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:46.414166 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:46.427345 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 01:59:46.488394 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:59:46.501056 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:59:46.626039 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:59:46.676419 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:59:46.707412 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:46.712005 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:46.716873 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:46.718847 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 01:59:46.720621 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:59:46.727732 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:46.735385 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:59:46.737607 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:46.737975 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:46.773247 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:59:46.800072 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:59:46.931231 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:59:46.987746 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:47.039016 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:47.041718 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:47.041714 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:47.048179 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:47.048535 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:47.054565 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:47.068033 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:47.366739 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:47.367005 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:47.370082 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:47.377189 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:47.382176 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:47.385296 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:47.394042 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:47.410271 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:47.691776 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:47.692827 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:47.693290 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:47.696393 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:47.708051 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:47.710508 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:47.712941 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 01:59:47.908640 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:48.017761 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:48.018110 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:48.018186 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:48.030083 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:48.030908 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:48.032655 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:48.055723 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:48.311189 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:48.343032 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:48.345644 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:48.348006 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:48.361934 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:48.366698 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:48.367566 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:48.368940 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 01:59:48.602336 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:59:48.671759 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:48.671862 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:48.678551 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:48.682229 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:48.688213 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:48.694782 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:48.694941 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:48.757213 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:59:48.785815 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:59:48.892814 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:59:48.921998 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:48.996232 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:48.997150 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:49.000468 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:49.009582 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:49.010577 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:49.022218 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:49.037272 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 01:59:49.096079 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:59:49.114493 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:59:49.135098 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:59:49.184456 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 01:59:49.192437 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:59:49.228065 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 01:59:49.228734 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:59:49.317894 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:59:49.319723 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:59:49.327842 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:49.328163 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:49.331369 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:49.331561 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:49.339629 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:49.346539 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:49.357394 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:49.381033 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240502 01:59:49.381807 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 01:59:49.440955 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:59:49.496302 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:59:49.659891 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:49.662236 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:49.666134 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:49.667945 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:49.670804 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:49.671769 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:49.678663 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:59:49.807451 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:49.976436 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:49.978729 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:49.983861 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:49.986794 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:49.997888 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:50.015718 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:50.016127 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:50.303642 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:50.304774 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:50.308688 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:50.312752 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:50.320655 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:50.321368 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:50.328227 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:50.347816 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:50.633683 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:50.634088 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:50.635154 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:50.646097 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:50.651069 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:50.651340 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:50.659355 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:50.774575 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 01:59:50.878250 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:59:50.957859 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:50.959322 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:50.960757 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:50.964037 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:50.975666 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:50.978744 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:50.980932 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:51.046689 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:59:51.119696 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:59:51.285437 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:51.290244 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:51.292631 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:51.293025 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:51.293440 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:51.298537 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:51.301270 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 01:59:51.383026 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 01:59:51.434166 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:59:51.597620 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:59:51.609817 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:51.617327 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:51.622848 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:51.631235 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:51.634217 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:51.634438 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:51.636920 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:59:51.639014 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:51.643239 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:59:51.685703 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:59:51.742610 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:59:51.773330 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:59:51.850322 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 01:59:51.886773 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:59:51.943773 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:51.945130 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:51.947209 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:51.947937 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:51.955462 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:51.955516 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:51.957119 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 01:59:52.123695 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:59:52.272629 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:52.274483 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:52.274513 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:52.279958 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:52.282290 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:52.282804 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:52.314363 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:52.364686 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:52.603919 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:52.604439 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:52.606741 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:52.607287 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:52.623553 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:52.628270 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:52.635830 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:52.933104 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:52.935819 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:52.936930 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:52.943480 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:52.949371 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:52.949899 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:52.950194 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:52.962721 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 01:59:53.221997 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:53.260813 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:53.263348 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:53.263398 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:53.270774 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:53.270794 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:53.274570 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:53.292811 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:53.502915 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:53.560395 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:53.582909 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:53.585019 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:53.585500 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:53.586979 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:53.587363 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:53.588372 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:53.595147 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 01:59:53.650565 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:53.740408 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:53.817840 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:53.897583 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:53.913579 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:53.915225 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:53.916684 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:53.917105 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:53.919746 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:53.932920 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:53.944782 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 01:59:53.991648 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:54.122215 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:54.124397 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:54.179286 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 01:59:54.247712 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 01:59:54.250236 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:54.256204 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:54.261868 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:54.269017 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:54.280344 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:54.296085 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:54.296882 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:54.329144 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 01:59:54.340391 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:59:54.468209 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:54.541823 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:59:54.595185 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 01:59:54.596166 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:54.613317 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:54.613641 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:54.615397 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:54.617808 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:54.619129 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 01:59:54.619315 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:59:54.620954 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:54.649142 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:54.942698 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:54.948130 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:54.948125 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:54.948357 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:54.958336 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:54.966912 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:54.974964 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:54.978372 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:55.276444 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:55.282043 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:55.284178 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:55.295634 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:55.297065 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:55.298720 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:55.299139 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:55.523985 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:55.628151 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:55.631103 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:55.634974 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:55.635133 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:55.647848 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:55.650205 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:55.662150 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:55.767021 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:55.980434 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:59:55.987481 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 01:59:55.993405 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:55.994557 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:55.995116 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:55.996171 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:56.004673 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:56.008015 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:56.014209 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:56.021692 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:56.220609 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:56.313678 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:56.321849 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:56.322362 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:56.325634 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:56.327240 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:56.327507 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:56.339403 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:56.356374 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:56.399647 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:56.420022 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 01:59:56.666471 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:56.671279 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:56.673188 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:56.673472 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:56.674034 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:56.674332 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:59:56.697930 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:59:56.698967 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:56.699230 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:56.707252 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:56.816443 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:57.005353 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:57.006210 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:57.006600 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:57.011771 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:57.014046 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:57.022279 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:57.025642 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 01:59:57.076215 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:57.226989 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:57.257788 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:59:57.325320 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:57.334626 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:57.334825 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:57.337109 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:57.344930 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:57.365327 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:57.366534 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:57.367851 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:57.677234 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:57.679908 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:57.680398 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:57.682797 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:57.690883 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:57.693441 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:57.695590 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 01:59:57.909241 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:58.011617 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:58.015179 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:58.015800 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:58.020351 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:58.022907 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:58.037035 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:58.045696 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 01:59:58.255604 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:58.349045 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:58.354893 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:58.362682 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:58.367134 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:58.373201 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:58.374181 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:58.378315 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 01:59:58.416229 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 01:59:58.505051 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:58.579363 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:58.688611 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:58.689532 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:58.693579 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:58.702900 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:58.705367 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:58.705677 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:58.707504 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:58.764509 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:58.843655 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 01:59:58.949952 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:58.969756 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:58.995532 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:59.013386 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:59.016479 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:59.016752 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:59.016829 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:59.023823 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:59.035516 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:59.055346 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:59.299124 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 01:59:59.317571 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:59.320132 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:59.345680 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:59.353623 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:59.355038 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:59.363066 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:59:59.363327 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:59:59.364224 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 01:59:59.364540 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 01:59:59.375643 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:59.396653 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 01:59:59.530200 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 01:59:59.604975 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:59:59.684774 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:59.688813 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:59.690740 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:59.701047 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:59:59.701331 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:59:59.702905 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 01:59:59.708968 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:59:59.709656 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 01:59:59.726222 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:59.738689 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:59:59.863605 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 02:00:00.014083 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:00.014992 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:00.016703 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:00.019009 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:00.033320 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:00.036666 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:00.046669 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:00.187314 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:00.348798 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:00.352175 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:00.356649 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:00.364746 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:00.367100 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:00.378103 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:00.378155 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:00.636699 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 02:00:00.681823 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:00.682858 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:00.683477 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:00.686868 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:00.689456 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:00.690798 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:00.711138 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:00.715031 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:00.898097 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:00:00.964746 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 02:00:01.023322 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:01.023721 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:01.026908 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:01.028621 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:01.034600 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:01.036003 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:01.045940 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:01.198453 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:00:01.211438 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:01.342692 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:01.350505 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:01.351918 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 02:00:01.354702 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:01.359573 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:01.360075 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:01.376650 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:01.380911 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:01.412799 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:00:01.611514 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:01.627075 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 02:00:01.672884 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:01.674175 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:01.674450 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:01.679998 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:01.689630 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:01.689985 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:01.703205 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:01.756753 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:00:01.776082 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:00:01.903152 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 02:00:01.988183 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:01.994346 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:01.998071 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:02.000655 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:02.004292 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:02.014775 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:02.015722 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:02.019730 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:02.316907 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:02.317700 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:02.317745 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:02.318017 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:02.319291 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:02.336621 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:02.339895 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:02.380633 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:00:02.471004 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:00:02.511915 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:02.646821 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:02.648351 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:02.653887 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:02.657649 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:02.664230 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:02.669620 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:02.679772 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:02.857826 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 02:00:02.964326 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:02.965507 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:02.966194 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:02.970237 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:02.988317 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:02.998651 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:03.001950 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:03.088755 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:03.239470 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 02:00:03.292358 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:03.299052 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:03.299677 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:03.303520 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:03.310364 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:03.313985 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:03.314567 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:03.376185 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:00:03.559532 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:03.621523 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:03.626659 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:03.632445 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:03.651134 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:03.653960 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:03.658190 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:03.663594 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:03.697204 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:00:03.795277 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 02:00:03.961499 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:03.963686 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:03.964800 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:03.966061 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:03.982740 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:03.984097 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:03.991003 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:04.012287 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:00:04.026927 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:00:04.136878 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:04.271744 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 02:00:04.288370 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:04.288401 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:04.290212 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:04.296270 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:04.298705 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:04.299208 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:04.299600 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:04.383922 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:00:04.425920 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:00:04.517431 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 02:00:04.601308 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:04.601338 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:04.604621 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:04.604858 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:04.613726 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:04.618518 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:04.618754 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:04.642266 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:04.674518 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:00:04.675056 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:00:04.797152 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:04.850560 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:00:04.862581 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 02:00:04.925278 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:04.927109 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:04.928354 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:04.931636 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:04.939823 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:04.941704 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:04.953297 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:05.217389 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:05.269649 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:05.271956 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:05.275794 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:05.284543 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:05.302634 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:05.307106 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:05.311511 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:05.573891 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:00:05.590577 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 02:00:05.614926 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:05.617286 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:05.623929 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:05.628772 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:05.629828 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:05.636031 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:05.650076 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:05.784188 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:05.942530 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:05.944532 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:05.945353 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:05.948693 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:05.950589 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:05.962193 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:05.965854 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:06.009516 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:00:06.128160 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:00:06.254235 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:06.272714 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:06.272881 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:06.273154 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:06.273423 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:06.280608 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:06.282883 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:06.302608 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:06.310966 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:00:06.325120 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:00:06.536458 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 02:00:06.590134 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:06.591413 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:06.595714 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:06.596864 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:06.597656 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:06.598742 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:06.598850 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:06.734253 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:06.735591 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:00:06.825585 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 02:00:06.917416 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:06.918275 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:06.920485 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:06.922016 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:06.922252 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:06.931927 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:06.946658 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:07.089514 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:00:07.201041 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:07.238535 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:07.240319 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:07.241431 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:07.244097 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:07.247819 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:07.248914 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:07.260442 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:07.359443 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:00:07.427170 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 02:00:07.553592 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:07.553717 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:07.559441 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:07.562822 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:07.566762 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:07.568228 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:07.583825 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:07.649209 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:07.883505 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:07.883934 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:07.885510 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:07.891168 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:07.892544 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:07.895628 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:07.913097 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:08.069908 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:00:08.128927 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:00:08.185118 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:08.211194 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:08.211572 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:08.214459 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:08.214515 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:08.216218 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:08.231384 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:08.237264 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:08.528208 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:08.531728 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:08.532186 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:08.536509 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:08.548527 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:08.549363 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:08.557082 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:08.621362 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:00:08.671204 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:00:08.672571 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:00:08.693153 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:00:08.747414 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:08.820117 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 02:00:08.851783 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:08.853695 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:08.858563 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:08.859781 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:08.873276 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:08.873497 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:08.875321 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:08.969347 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:00:09.125087 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:00:09.170740 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:09.176373 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:09.176898 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:09.177549 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:09.178453 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:09.179801 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:09.182579 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:09.186554 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:09.466300 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 02:00:09.510435 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:09.511596 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:09.514931 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:09.517771 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:09.524420 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:09.531922 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:09.536876 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:09.605320 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:09.661937 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:00:09.741243 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:09.802768 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 02:00:09.854189 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:09.856057 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:09.860805 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:09.861336 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:09.865684 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:09.870954 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:09.888451 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:10.008879 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:10.068504 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 02:00:10.094424 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:00:10.133189 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:10.181485 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:10.186946 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:10.188877 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:10.189296 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:10.189497 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:10.191944 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:10.192305 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:10.423681 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 02:00:10.499194 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:10.500659 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:10.504719 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:10.506043 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:10.514571 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:10.514916 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:10.528903 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:10.584115 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:10.724086 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 02:00:10.820704 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:10.823760 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:10.827939 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:10.834672 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:10.846310 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:10.846447 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 02:00:10.846890 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:10.854970 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:10.936153 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:00:11.049168 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:11.084641 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:00:11.098315 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 02:00:11.144351 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:11.147240 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:11.150066 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:11.155642 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:11.161263 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:11.162197 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:11.170341 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:11.329484 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:00:11.343314 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 02:00:11.488143 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:11.489161 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:11.491015 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:11.491999 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:11.492972 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:11.493211 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:11.497646 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:11.519560 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:11.727018 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 02:00:11.815804 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:11.816196 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:11.816890 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:11.821141 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:11.830518 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:11.834792 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:11.838912 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:11.929128 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:00:12.035415 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:12.135610 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:12.137974 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:12.139427 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:12.139629 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:12.152269 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:12.153509 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:12.153748 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:12.377034 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:00:12.437036 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:12.457057 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:12.465432 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:12.466095 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:12.467890 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:12.468730 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:12.468842 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:12.474164 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:12.549861 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 02:00:12.783509 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:12.785710 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:12.791002 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:12.794258 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:12.798802 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:12.803742 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:12.804894 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:12.857342 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:13.110359 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:13.110805 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:13.112912 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:13.114495 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:13.122547 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:13.127390 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:13.128684 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:13.143735 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:00:13.266899 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:00:13.272982 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:00:13.398447 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:13.405149 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 02:00:13.438908 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:13.444402 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:13.450971 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:13.460098 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:13.463724 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:13.475275 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:13.475385 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:13.574474 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:00:13.606730 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 02:00:13.765924 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:13.767985 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:13.770377 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:13.779557 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:13.781339 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:13.787112 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 02:00:13.795969 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:13.797672 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:13.825131 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:14.060425 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 02:00:14.101392 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:14.101486 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:14.102367 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:14.109140 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:14.111718 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:14.117293 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:14.124320 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:14.196038 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:00:14.215097 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:14.425784 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:14.426584 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:14.428508 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:14.432157 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:14.442155 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:14.453019 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:14.458005 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:14.482599 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:00:14.531522 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:00:14.653272 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 02:00:14.751058 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:14.751058 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:14.751996 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:14.752357 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:14.752759 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:14.754696 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:14.768675 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:14.793701 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:14.991717 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:00:15.048491 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 02:00:15.084059 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:15.091820 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:15.094117 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:15.096560 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:15.096786 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:15.107192 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:15.107533 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:15.167062 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:00:15.185644 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:15.242273 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:00:15.400977 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:00:15.568802 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:00:15.597579 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:15.663600 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:00:15.964829 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:00:16.144142 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:00:16.147912 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:16.266842 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:00:16.357628 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 02:00:16.393337 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:16.398504 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:16.399868 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:16.400457 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:16.407032 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:16.414577 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:16.425283 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:16.545960 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:00:16.621529 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:16.730577 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:16.731022 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:16.732295 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:16.732933 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:16.734366 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:16.734742 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:16.738876 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 02:00:16.746328 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:16.787173 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:00:17.032419 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 02:00:17.052008 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:17.053346 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:17.054625 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:17.061614 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:17.066571 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:17.070770 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:17.072935 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:17.177188 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:17.387648 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:17.391172 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:17.391185 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:17.391762 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:17.399518 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:17.406518 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:17.410573 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:17.543483 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 02:00:17.720669 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:17.722245 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:17.729799 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:17.731515 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:17.741164 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:17.742950 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:17.756644 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:17.792435 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:17.854449 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:00:17.913825 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:00:17.915374 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 02:00:18.053081 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:18.059620 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:18.067679 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:18.069937 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:18.075647 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:18.076614 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:18.093212 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:18.323769 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:18.367444 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 02:00:18.401602 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:18.401672 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:18.403740 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:18.404755 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:18.405162 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:18.427438 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:18.445122 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:18.575851 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:00:18.719645 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:18.740003 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:18.742679 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:18.745085 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:18.751931 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:18.759178 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 02:00:18.759899 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:18.759898 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:18.777007 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:18.921562 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 02:00:19.075218 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:19.080433 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:19.080906 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:19.083012 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:19.090154 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:19.095854 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:19.105042 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:19.122467 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:19.272228 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:00:19.293149 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:00:19.332386 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 02:00:19.413048 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:19.418975 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:19.420565 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:19.422608 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:19.432593 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:19.439386 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:19.446897 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:19.646878 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:00:19.695318 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:00:19.736688 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:19.750277 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:19.750725 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:19.765523 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:19.769863 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:19.770037 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:19.775889 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:19.791416 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:19.911954 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 02:00:20.088919 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:20.092969 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:20.093111 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:20.102558 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:20.103346 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:20.111270 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:20.112728 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:20.216142 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:20.216894 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:20.244647 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:00:20.263571 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:00:20.264317 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:00:20.289126 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:00:20.419376 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:20.421907 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:20.431375 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:20.440712 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:20.442981 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:20.443374 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:20.443727 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 02:00:20.444977 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:20.746148 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:20.748857 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:20.753350 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:20.759373 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:20.766997 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:20.779162 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:20.779637 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:20.815804 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:20.846882 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:00:20.973157 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:00:21.053282 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 02:00:21.077020 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:21.079921 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:21.080186 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:21.080607 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:21.093714 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:21.095428 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:21.097048 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:21.403707 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:21.407312 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:21.411115 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:21.420073 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:21.422360 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:21.427225 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:21.428712 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:21.430907 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:21.431934 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 02:00:21.741868 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:21.741925 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:21.754599 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:21.757376 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:21.757695 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:21.759094 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:21.774760 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:21.825704 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:00:21.978354 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:00:21.993518 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:22.009984 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 02:00:22.072094 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:22.077359 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:22.081794 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:22.083804 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:22.084051 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 02:00:22.084113 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:22.085259 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:22.110097 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:22.410670 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:22.410842 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:22.412247 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:22.419358 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:22.428360 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 02:00:22.429224 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:22.435736 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:22.438899 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:22.524578 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:22.741441 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:22.746170 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:22.747838 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:22.750553 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:22.758561 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:22.759785 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:22.770990 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:22.891570 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:00:22.908413 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:00:22.968591 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:22.988940 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 02:00:23.075560 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:23.084959 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:23.086109 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:23.086788 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:23.087646 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:23.104255 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:23.106343 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:23.112915 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:00:23.386183 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:00:23.388718 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:23.417027 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:23.417563 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:23.419101 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:23.419651 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:23.424168 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:23.434444 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:23.442550 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:23.741693 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:23.743170 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:23.746423 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:23.746577 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:23.766242 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:23.767017 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:23.768107 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:23.793334 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:23.854254 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:00:23.912607 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 02:00:24.077227 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:24.078027 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:24.087172 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:24.087180 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:24.091512 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:24.091965 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:24.103202 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:24.274168 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:24.274722 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 02:00:24.425489 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:24.425813 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:24.439744 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:24.440007 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:24.454020 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:24.454640 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:24.468863 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:24.619283 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 02:00:24.781112 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:24.782142 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:24.783277 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:24.785230 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:24.786602 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 02:00:24.795392 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:24.806329 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:24.807250 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:24.815160 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:00:24.815919 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:00:24.837677 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:00:24.875582 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:25.107154 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:25.107944 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:25.112655 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:25.114611 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:25.120641 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:25.124497 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:25.127974 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:25.315115 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:25.315686 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:25.431252 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:25.431842 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:25.445392 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:25.452709 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:00:25.455235 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:25.456979 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:25.468684 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:25.471383 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:25.472184 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:00:25.538688 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:00:25.565750 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:00:25.611665 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:00:25.686053 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:00:25.721499 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:25.768923 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:25.769505 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:25.773612 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:25.776538 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:25.779950 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:25.798362 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:25.806736 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:25.901059 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 02:00:26.098948 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:26.099961 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:26.102573 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:26.109226 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:26.111346 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:26.114094 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:26.118553 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:26.261240 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:26.427574 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:26.433912 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 02:00:26.435786 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:26.440239 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:26.441464 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:26.449258 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:26.449717 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 02:00:26.469686 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:26.470604 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:26.753365 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 02:00:26.774590 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:26.775014 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:26.776731 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:26.784857 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:26.790532 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:26.790717 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:26.813031 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:26.828022 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:26.862493 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:00:27.073796 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 02:00:27.105120 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:27.108068 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:27.117683 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:27.126439 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:27.128860 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:27.129808 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:27.151295 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:27.353078 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:27.441679 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:27.444252 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:27.445509 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:27.446287 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:27.457012 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:27.458353 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:27.482587 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:27.580217 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:00:27.664947 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 02:00:27.778028 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:27.781296 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:27.783006 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:27.788700 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:27.789074 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 02:00:27.791486 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:27.802575 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:27.818980 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:27.936935 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:00:27.970374 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:28.110357 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:28.114177 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:28.121945 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:28.123306 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:28.123962 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:28.150969 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:28.152737 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:28.172669 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:00:28.259781 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 02:00:28.444160 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:28.448668 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:28.454314 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:28.455006 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:28.457108 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:28.461277 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:28.466387 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 02:00:28.467551 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:28.496479 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:28.650691 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 02:00:28.775300 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:28.776074 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:28.782567 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:28.788035 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:28.799011 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:28.800180 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:28.810225 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:29.077663 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:00:29.084506 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:00:29.094894 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:29.108853 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:29.108871 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:29.119174 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:29.119210 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:29.119737 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:29.120627 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:29.137555 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:29.390244 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:00:29.400414 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 02:00:29.438611 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:29.442871 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:29.449115 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:29.455367 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:29.456444 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:29.480737 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:29.480752 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:29.587391 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:29.774096 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:29.777971 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:29.781626 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:29.795243 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:29.795253 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:29.798729 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:29.798784 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:29.902511 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:00:29.903124 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:00:30.066470 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:30.110426 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:30.114267 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:30.118598 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:30.124349 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:30.127184 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:30.140823 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:30.142701 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:30.194617 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:00:30.336268 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 02:00:30.447134 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:30.449474 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:30.450623 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:30.454759 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:30.471150 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 02:00:30.475617 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:30.478260 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:30.486302 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:30.487221 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:30.488912 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:30.598834 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:00:30.718824 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:30.719450 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:00:30.767792 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 02:00:30.783385 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:30.785656 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:30.787410 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:30.789824 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:30.790796 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:30.811939 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:30.813470 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:30.824038 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 02:00:31.095204 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:31.112039 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:31.112427 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:31.114048 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:31.117180 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:31.129981 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:31.140100 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:31.141101 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:31.416592 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 02:00:31.441361 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:31.441920 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 02:00:31.443368 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:31.446817 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:31.447197 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:31.451215 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:31.468395 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:31.478173 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:31.559062 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:00:31.636258 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:31.758457 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 02:00:31.779081 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:31.780087 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:31.782994 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:31.788650 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:31.790180 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:31.790982 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:31.823518 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:31.938136 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 02:00:32.112285 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:32.114058 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:32.124516 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:32.135023 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:32.136230 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:32.136224 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:32.137737 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:32.221585 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:32.281502 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 02:00:32.441491 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:32.443171 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:32.452330 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:32.453119 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:32.454046 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:32.472797 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:32.485146 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:32.603781 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:32.782692 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:32.787174 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:32.790766 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:32.791863 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:32.792011 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:32.792608 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:32.796607 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:32.820528 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:00:32.854009 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:00:32.946794 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:00:33.020923 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:00:33.057328 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 02:00:33.114014 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:33.114478 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:33.120713 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:33.123442 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:33.126863 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:33.135403 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:33.143746 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:33.144174 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:33.447469 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:33.447916 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:33.453018 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:33.456832 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:33.462750 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:33.476226 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:33.476799 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:33.580493 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:33.620565 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 02:00:33.772336 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:33.781011 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:33.785742 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:33.785920 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:33.787864 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:33.806411 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:33.806548 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:33.903402 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:00:33.922302 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:00:34.027617 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:34.049551 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 02:00:34.108865 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:34.110627 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:34.114980 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:34.117285 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:34.125325 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:34.131033 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:34.157779 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:34.299615 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:00:34.385737 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:00:34.434119 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:34.451634 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:34.452543 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:34.458313 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:34.463369 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:34.466632 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:34.479331 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:34.490119 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:34.682734 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 02:00:34.784200 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:34.789644 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:34.793568 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:34.795498 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:34.797750 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:34.798146 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:34.811472 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:34.828773 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:34.953168 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 02:00:35.126338 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:35.128185 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:35.129374 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:35.130154 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:35.140868 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:35.143282 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:35.143671 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:35.369661 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:35.382151 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 02:00:35.457091 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:35.460911 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:35.461086 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:35.464926 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:35.465373 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:35.468703 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:35.484565 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:35.547019 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:00:35.646008 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:00:35.776784 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:35.777606 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:35.792635 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:35.793823 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:35.794061 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:35.802915 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:35.816743 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:35.819521 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 02:00:35.820016 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:35.822448 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:35.856122 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:00:35.985039 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:00:36.030815 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:00:36.131101 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:36.137796 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:36.138306 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:36.146631 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:36.149967 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 02:00:36.156726 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:36.157543 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:36.158043 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:36.181175 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:36.257591 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 02:00:36.468541 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:36.473459 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:36.475251 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:36.477802 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 02:00:36.481161 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:36.486601 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:36.497584 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:36.498080 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:36.522799 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:00:36.560761 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:36.632519 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:00:36.744639 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 02:00:36.810587 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:36.811851 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:36.814714 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:36.815533 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:36.819579 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:36.822320 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:36.836629 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:36.961450 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:37.143504 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:37.145505 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:37.147924 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 02:00:37.154918 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:37.157626 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:37.159132 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:37.159451 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:37.160159 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:37.482751 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:37.482898 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:37.484477 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:37.485285 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:37.485770 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:37.503198 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:37.503382 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:37.554183 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:37.808434 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:37.812880 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:37.813916 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:37.823405 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:37.826313 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:37.827011 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:37.844604 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:37.880196 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:00:38.084229 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:00:38.087514 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 02:00:38.143383 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:38.143961 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:38.151496 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:38.155370 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:38.159991 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:38.165042 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:38.166033 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:38.172021 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:38.359930 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:00:38.430529 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 02:00:38.472035 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:38.472147 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:38.479895 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:38.482705 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:38.486570 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:38.489277 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:38.515290 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:38.569025 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:00:38.598908 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:38.805922 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:38.809338 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:38.810443 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:38.812726 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:38.813400 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:38.835058 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:38.835740 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:38.841367 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:00:38.922925 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:00:39.071445 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:00:39.084437 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:39.137737 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:39.144338 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:39.145623 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:39.146744 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:39.147673 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:39.155277 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:39.164762 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:39.284411 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:00:39.294878 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 02:00:39.474447 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:39.478268 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:39.482040 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:39.485906 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:39.487051 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:39.487460 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:39.504212 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:39.646229 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:39.812340 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:39.816093 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:39.818181 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:39.826473 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:39.828758 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:39.829336 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:39.843556 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:39.884364 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:00:39.954353 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 02:00:40.143728 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:40.144701 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:40.146822 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:40.147490 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:40.155365 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:40.162328 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:40.179090 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:40.201485 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:40.390231 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 02:00:40.477828 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:40.481528 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:40.483908 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:40.484320 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:40.486593 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:40.495165 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:40.497778 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:40.634029 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:00:40.662180 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:00:40.775394 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:00:40.775693 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:40.804929 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:40.806030 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:40.806622 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:40.806900 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:40.817235 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:40.817235 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:40.827977 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:40.912384 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:00:41.043118 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:00:41.091197 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:41.091919 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 02:00:41.136454 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:41.140481 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:41.140655 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:41.147257 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:41.156188 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:41.157686 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:41.164917 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 02:00:41.174600 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:41.245666 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:41.309680 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:41.465641 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:41.468255 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:41.476171 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:41.477391 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:41.478919 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:41.483737 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:41.505627 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:41.536924 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 02:00:41.804054 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:41.804490 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:41.808274 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:41.808833 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:41.817567 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:41.824532 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:41.829833 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:41.842902 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:00:41.845557 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:41.916201 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:00:41.954663 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 02:00:42.135445 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:42.143247 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:42.144196 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:42.146601 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:42.146593 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:42.153102 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:42.160641 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:42.450635 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:42.467898 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:42.470820 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:42.477756 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:42.478569 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:42.481482 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:42.489115 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:42.498610 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:42.576010 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 02:00:42.802819 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:42.804378 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:42.807237 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:42.808912 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:42.818382 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:42.825644 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:42.839152 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:42.840878 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:42.951257 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:00:43.083956 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:00:43.113832 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 02:00:43.138778 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:43.140100 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:43.140704 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:43.141389 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:43.144115 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:43.147081 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:43.148298 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:43.303575 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:00:43.394100 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:43.451319 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 02:00:43.469559 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:43.471796 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:43.473222 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:43.474962 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:43.482556 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:43.490808 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:43.491956 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:43.652427 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:00:43.675647 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:00:43.705829 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 02:00:43.801837 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:43.802778 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:43.805763 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:43.805827 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:43.811728 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:43.813961 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:43.826956 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:43.845860 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:44.137611 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:44.142601 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:44.145308 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:44.147552 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:44.153170 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:44.173134 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:44.174552 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:44.296577 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:00:44.297137 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:44.392680 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 02:00:44.476294 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:44.478708 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:44.479266 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:44.488242 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:44.490971 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:44.494587 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:44.496680 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:44.620719 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:00:44.726931 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:44.805651 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:44.809201 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:44.813406 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:44.814963 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:44.820394 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:44.822445 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:44.833566 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:45.036650 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240502 02:00:45.037446 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 02:00:45.141538 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:45.144466 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:45.146762 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:45.147917 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:45.158991 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:45.159300 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:45.174607 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:45.207717 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:45.326634 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:00:45.472174 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 02:00:45.475446 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:45.482725 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:45.484916 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:45.485538 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:45.485684 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:45.487653 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:45.488132 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:45.563463 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:00:45.632532 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:00:45.775054 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:45.806073 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:45.809944 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:45.810356 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:45.811127 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:45.811683 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:45.827446 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 02:00:45.831077 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:45.836891 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:46.105500 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:46.106220 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:00:46.119309 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 02:00:46.138747 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:46.142154 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:46.143440 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:46.154151 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:46.157147 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:46.162950 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:46.169685 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:46.206971 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:46.260062 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:00:46.342085 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 02:00:46.472776 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:46.475435 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:46.475653 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:46.476374 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:46.477562 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:46.488243 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 02:00:46.501366 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:46.516685 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:46.598502 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:00:46.676471 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:00:46.757799 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:46.816882 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:46.818753 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:46.819075 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:46.822212 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:46.823271 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:46.835777 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:46.840749 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:47.058352 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 02:00:47.143111 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:47.143931 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:47.145959 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:47.146525 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:47.150440 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:47.172894 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:47.181682 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:47.201036 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:47.469766 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:47.473034 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:47.474781 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:47.475078 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:47.477913 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:47.507661 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:47.508751 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:47.571918 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:00:47.758473 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:47.802029 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 02:00:47.804049 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:47.805800 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:47.807396 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:47.810969 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:47.813472 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:47.821882 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:47.826594 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:47.932041 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 02:00:48.130167 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:48.137042 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:48.137329 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:48.137640 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:48.150082 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:48.151970 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:48.168201 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:48.185261 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:00:48.217353 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:00:48.247965 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:48.448942 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 02:00:48.469350 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:48.471237 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:48.472295 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:48.473107 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:48.475833 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:48.485224 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:48.487531 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:48.550889 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:00:48.647567 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:00:48.755110 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 02:00:48.803313 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:48.807540 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:48.808264 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:48.810876 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:48.814797 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:48.814944 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:48.818790 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:48.826900 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:49.136197 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:49.136708 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 02:00:49.137614 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:49.141460 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:49.150604 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:49.157212 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:49.159320 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:49.168628 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:49.267498 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:49.390544 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:00:49.403656 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 02:00:49.468704 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:49.468783 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:49.471652 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:49.475006 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:49.481266 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:49.495252 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:49.500766 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:49.578042 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:00:49.792951 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:49.795974 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:49.801954 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:49.802925 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:49.809312 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:49.809962 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:49.819067 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:49.819722 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:50.122934 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:50.124126 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:50.129969 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:50.130277 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:50.134683 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 02:00:50.152119 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:50.157249 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:50.157685 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:50.173125 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:50.193156 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:00:50.244617 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 02:00:50.455979 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:50.461177 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:50.461459 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:50.462379 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:50.466024 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:50.472508 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:50.483390 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:50.595182 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:00:50.627393 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:00:50.709620 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:50.784950 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:50.785050 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:50.787688 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:50.796435 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:50.796473 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:50.799733 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:50.800220 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:50.922112 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:00:50.926504 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:00:51.032181 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 02:00:51.109041 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:51.118261 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:51.119809 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:51.120682 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:51.121982 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:51.135375 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:51.136729 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:00:51.151435 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:51.263103 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:00:51.274971 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:51.382177 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 02:00:51.454219 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:51.460027 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:51.461620 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:51.462553 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:51.463572 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:51.487356 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:51.489602 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:51.570051 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:00:51.570885 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:00:51.592062 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 02:00:51.793579 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:51.796267 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:51.799749 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:51.801213 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:51.806023 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:51.811784 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:51.820379 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:51.853825 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:52.129673 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:52.131515 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:52.133205 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:52.134577 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:00:52.135167 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:52.154423 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 02:00:52.160733 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:52.161037 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:52.161602 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:52.262171 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:52.468592 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:52.472146 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:52.476858 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:52.478569 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:52.486637 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:52.486824 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:52.502220 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:52.564071 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:00:52.651417 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:00:52.798357 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:52.802603 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:52.806495 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:52.807112 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:52.808459 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:52.818619 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:52.824282 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:52.835912 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:53.060928 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 02:00:53.139508 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:53.142011 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:53.143620 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:53.144479 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:00:53.146981 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:53.158370 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:53.177326 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:53.178926 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:53.327292 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:53.336724 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:53.370846 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:00:53.475121 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:53.477360 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:53.487890 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:53.488745 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:53.490571 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:53.490908 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:53.491861 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:53.526962 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:00:53.804514 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:53.804663 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:53.805449 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:53.812194 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:53.814559 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:53.816169 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:53.823376 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:53.943643 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:54.111687 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:54.127815 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:00:54.130481 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:54.131546 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:54.136771 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:54.137990 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:54.145246 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:54.145450 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:54.154976 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:54.194151 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:00:54.312947 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:00:54.462472 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:54.462651 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:54.463871 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:54.464474 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:54.465946 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:54.470268 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:54.489733 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:54.558483 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:54.723822 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:54.756273 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:00:54.787043 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:54.792320 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:54.793902 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:54.800498 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:54.803452 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:54.808158 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:54.811806 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:54.966585 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:55.113720 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:55.117390 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:55.122361 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:55.124926 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:55.125367 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:55.145056 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:55.147797 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:55.262511 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:00:55.286060 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:55.403301 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:55.445513 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:55.450230 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:55.452860 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:55.465626 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:00:55.468561 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:00:55.473909 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:55.479637 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:55.493978 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:55.786721 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:55.791815 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:55.797993 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:55.800087 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:55.800609 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:55.803611 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:55.803804 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:55.890990 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:55.908394 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:55.911826 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:55.916898 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:56.112717 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:56.118111 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:56.122563 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:56.124665 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:56.153097 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:56.153527 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:56.155604 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:56.271724 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:56.339937 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:56.465217 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 02:00:56.470089 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:56.470342 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:56.472507 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:56.473134 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:56.488082 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:56.489898 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:56.513211 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:56.538772 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:00:56.695001 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:56.719024 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:00:56.736456 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:56.767613 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:56.773487 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 02:00:56.812479 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:56.812938 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:56.813624 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:56.819655 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:56.821694 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:56.822016 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:56.841902 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:56.947760 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:57.139508 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 02:00:57.141693 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:57.142062 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:57.142640 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:57.145668 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:57.149034 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:57.152334 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:57.177584 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:57.263515 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:00:57.284225 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:57.473776 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:57.476351 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:57.481817 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:57.482558 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:57.491020 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:57.494782 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:57.500792 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:57.669895 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 02:00:57.793236 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:57.794981 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:57.804242 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:57.809154 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:57.813721 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:57.820811 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:57.822024 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:00:57.900532 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:58.063490 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 02:00:58.129407 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:58.133250 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:58.142547 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:58.142750 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:58.143687 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:58.151737 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:58.152477 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:58.183383 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:58.403334 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:58.464973 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:58.471607 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:58.473711 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:58.473997 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:58.474210 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:58.482280 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:58.504537 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:58.511162 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:58.521359 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:58.783735 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:58.785815 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:58.796414 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:58.798329 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:58.802419 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:58.804442 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:58.809157 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:58.810353 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:58.837216 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:00:59.014627 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 02:00:59.133744 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:59.134518 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:59.141213 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:59.142127 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:59.142982 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:59.144974 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:59.163833 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:59.176357 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:59.207562 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:59.255218 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:00:59.422051 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:00:59.454916 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 02:00:59.460042 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:59.465200 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:59.468317 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:59.469377 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:59.482743 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:59.483119 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:59.497013 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:00:59.794077 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:59.798296 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:59.798346 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:59.798609 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:00:59.806330 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:59.809015 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:00:59.826143 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:00:59.839323 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:00.001011 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 02:01:00.139210 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:00.139565 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:00.142678 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:00.143000 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:00.152446 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:00.152782 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 02:01:00.165088 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:00.182179 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:00.217007 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:00.274776 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 02:01:00.475268 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:00.478250 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:00.478696 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:00.480173 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:00.482610 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:00.482966 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:00.483443 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:00.604511 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:01:00.691788 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:00.801959 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:00.809029 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:00.812083 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:00.812844 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:00.815943 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:00.818586 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:00.818867 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:00.878122 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:01:01.110855 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 02:01:01.141803 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:01.143229 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:01.145328 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:01.147637 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:01.151108 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:01.161243 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:01.170186 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 02:01:01.170693 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:01.228861 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:01:01.263912 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:01.405097 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:01:01.468277 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:01.473677 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:01.477350 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:01.477622 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:01.479988 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:01.481201 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:01.488121 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:01.629052 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:01:01.719511 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:01:01.720213 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:01:01.754683 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:01.755419 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:01:01.762486 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:01.800096 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:01.802464 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:01.805048 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:01.809453 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:01.824669 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:01.835775 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:01.847307 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:01.893148 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:01:02.125211 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 02:01:02.145581 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:02.148563 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:02.150900 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:02.151214 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:02.152374 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:02.162825 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:02.169155 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:02.209918 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:02.474510 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:02.474838 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:02.475875 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:02.482004 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:02.482304 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:02.498525 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:02.500118 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:02.662487 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:02.799803 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 02:01:02.805675 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:02.807061 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:02.808830 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:02.820189 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:02.820432 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:02.822139 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:02.824352 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:02.932392 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:01:02.986321 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:01:03.057139 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:03.135825 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:03.140551 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:03.142285 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:03.146737 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:03.146804 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:03.166496 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:03.168756 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:03.467599 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:03.468158 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:03.470013 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:03.472970 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:03.476397 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 02:01:03.487751 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:03.495121 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:03.499469 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:03.571308 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:03.591120 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:01:03.775187 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 02:01:03.803409 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:03.805614 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:03.807291 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:03.807503 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:03.815325 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:03.823465 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:03.831374 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:03.951313 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:01:04.103909 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:04.132251 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 02:01:04.135327 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:04.142455 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:04.145133 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:04.146582 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:04.148145 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:04.156806 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 02:01:04.166558 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:04.169754 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:04.347915 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:01:04.380288 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 02:01:04.467972 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:04.470420 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:04.470858 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:04.473327 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:04.479274 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 02:01:04.484405 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:04.484558 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:04.486142 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:04.501003 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:04.795065 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:04.795882 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:04.802570 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:04.802840 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:04.803025 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:04.825109 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:04.832091 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:04.900943 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:05.125001 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:05.134796 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:05.135254 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:05.138805 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:05.141351 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:05.143848 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:05.161510 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:05.253687 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:01:05.344447 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 02:01:05.466110 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:05.467769 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:05.468716 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:05.470868 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:05.470899 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:05.476891 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:05.478571 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:05.478600 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:05.596400 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:01:05.621362 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 02:01:05.793712 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:05.800047 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:05.800846 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:05.801153 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:05.807102 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:05.809392 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:05.814584 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:05.871526 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:01:05.884178 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:06.134820 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:06.138301 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:06.139523 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:06.145195 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:06.147197 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:06.150576 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:06.160128 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:06.256837 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:01:06.334455 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:01:06.430335 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:01:06.450709 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:06.452692 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:06.456598 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:06.462110 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:06.467248 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:06.468866 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:06.469449 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:06.478724 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:06.482036 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:06.643375 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:01:06.746570 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:01:06.747288 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:01:06.761911 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 02:01:06.795608 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:06.797914 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:06.802593 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:06.804092 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:06.810256 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:06.810431 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:06.810635 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:06.865676 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:06.971284 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:06.971959 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 02:01:07.132874 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:07.135044 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:07.140133 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:07.140866 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:07.141992 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 02:01:07.145742 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:07.166507 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:07.174841 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:07.269043 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:07.464380 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:07.471599 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:07.472251 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:07.473803 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:07.475574 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:07.475854 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:07.486315 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:07.734436 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 02:01:07.789458 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:07.792516 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:07.795341 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:07.795917 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:07.802624 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:07.815115 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:07.822966 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:07.870621 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:07.921180 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:01:07.929895 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 02:01:08.115290 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:08.118733 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:08.122113 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:08.124581 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:08.129156 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:08.139261 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:08.146955 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:08.278200 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:08.296456 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 02:01:08.440017 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:08.443593 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:08.450194 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:08.455421 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:08.459332 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:08.461016 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:08.467761 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:08.734341 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:08.772137 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:08.775003 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:08.783628 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:08.784078 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:08.786825 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:08.793658 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:08.802650 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:08.831777 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:01:08.922590 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:01:08.930230 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:01:09.015537 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 02:01:09.108633 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:09.111125 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:09.112792 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:09.114521 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:09.116089 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:09.117043 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:09.141774 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:09.272270 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:01:09.277166 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:09.336643 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:01:09.421063 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 02:01:09.443974 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:09.450749 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:09.451076 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:09.452607 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:09.461849 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:09.464504 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:09.476389 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:09.592598 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 02:01:09.777580 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:09.779458 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:09.781203 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:09.790107 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:09.792941 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:09.803682 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:09.805465 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:09.808496 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:09.960947 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 02:01:10.108914 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:10.111253 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:10.111395 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:10.112231 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:10.122740 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:10.129710 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:10.135035 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:10.254066 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:10.379534 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:01:10.404476 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240502 02:01:10.437784 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:10.439975 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 02:01:10.440822 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:10.444556 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:10.453981 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:10.454939 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:10.471120 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:10.481871 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:10.778329 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:10.780438 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:10.784417 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:10.790233 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:10.790738 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:10.794238 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:10.805253 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:10.818464 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:01:10.830973 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:11.111225 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:11.111740 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:11.113518 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:11.126602 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:11.127408 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:11.144363 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:11.158515 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:11.369843 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:01:11.417770 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:01:11.431628 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:01:11.432569 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:11.448593 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 02:01:11.450901 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:11.454288 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:11.456229 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:11.458046 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:11.472676 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:11.482585 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:11.489902 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:11.705152 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:01:11.781131 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 02:01:11.784574 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:11.784986 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:11.785414 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:11.789352 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:11.790339 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:11.805794 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:11.812964 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:11.817749 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:01:11.818645 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:11.859611 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:01:11.860260 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:01:12.017307 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 02:01:12.111198 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:12.116427 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:12.116708 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:12.124397 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:12.127319 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:12.128455 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:12.128592 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:12.290151 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:12.316361 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 02:01:12.440235 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:12.440855 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:12.450486 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:12.453020 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:12.453518 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:12.456337 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:12.469175 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:12.564010 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 02:01:12.766923 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:12.771824 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:12.775301 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:12.777199 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:12.778074 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:12.789912 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 02:01:12.793535 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:12.794044 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:12.796566 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:12.801599 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 02:01:12.913791 6632 maintenance_manager.cc:419] P 8f0a4860df0c4e4a97901c87e80d6420: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033377
I20240502 02:01:12.933588 6631 maintenance_manager.cc:643] P 8f0a4860df0c4e4a97901c87e80d6420: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.019s user 0.014s sys 0.003s Metrics: {"bytes_written":20196,"cfile_init":1,"dirs.queue_time_us":257,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1400,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":451,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":189,"threads_started":2,"wal-append.queue_time_us":213}
I20240502 02:01:12.934581 6632 maintenance_manager.cc:419] P 8f0a4860df0c4e4a97901c87e80d6420: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7952 bytes on disk
I20240502 02:01:12.935189 6631 maintenance_manager.cc:643] P 8f0a4860df0c4e4a97901c87e80d6420: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 02:01:13.100837 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:13.103552 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:13.103994 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:13.105337 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:13.105350 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:13.107918 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:13.135425 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:13.229945 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:01:13.401192 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:13.438936 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:13.443923 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:13.446321 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:13.446615 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:13.460398 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:13.463214 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:13.463263 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:13.598007 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 02:01:13.769502 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:13.770653 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:13.772287 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:13.779621 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:13.782850 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:13.800228 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:13.801884 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:13.937036 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:01:13.953683 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:14.091845 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:14.092698 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:14.098508 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:14.098794 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:14.104022 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:14.106623 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:01:14.106659 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 02:01:14.120453 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:14.137957 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:14.222348 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:01:14.277650 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 02:01:14.434281 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:14.434322 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:14.440806 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:14.443574 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:14.449116 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:14.452924 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:14.457317 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:14.470965 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:14.503772 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:01:14.530144 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:01:14.759636 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:14.761415 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:14.761983 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:14.774960 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:14.783445 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:14.788137 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:14.788419 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:14.866310 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:14.994989 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:01:15.029233 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:01:15.094038 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:15.095896 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:15.099120 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:15.103662 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:15.106565 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:15.106972 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:15.107918 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:15.370561 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:15.421265 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:01:15.423224 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:15.424780 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:15.426291 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:15.431803 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:15.440701 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:15.451912 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:15.453271 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:15.602033 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:01:15.602677 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 02:01:15.751147 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:15.751435 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:15.754705 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:15.762894 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:15.763679 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:15.764672 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:15.773630 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:15.975327 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:16.081620 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:16.084873 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:16.085274 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:16.086637 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:16.088884 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:16.091390 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:16.117022 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:16.136390 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:01:16.413204 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:16.414465 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:16.414646 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:01:16.416882 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:16.420641 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:16.426868 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:16.428843 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:16.431321 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:16.443073 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:16.482292 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:01:16.688546 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:01:16.730942 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:16.734529 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:16.734818 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:16.738651 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:16.742440 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:16.755290 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:16.762596 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:16.770365 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:01:16.861210 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:01:16.991519 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:16.992256 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:17.048522 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:01:17.065991 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:17.071422 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:17.071797 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:17.075354 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:17.077896 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:17.086885 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:17.103178 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:17.171514 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:01:17.275286 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:01:17.375964 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:01:17.402088 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:17.409446 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:17.410045 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:17.410991 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:17.411087 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:17.411911 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:17.438879 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:17.455858 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:17.456475 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:01:17.571983 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:17.733726 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:17.734984 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:17.736308 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:17.737330 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:17.743539 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:17.744725 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:17.745736 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:18.027537 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:18.058804 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:18.061161 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:18.062508 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:18.069365 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:18.075788 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:18.077195 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:18.078130 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:18.166023 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:01:18.191363 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:01:18.232125 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling FlushMRSOp(a66b932d884e4df8a78841103d88ded0): perf score=0.033344
I20240502 02:01:18.257655 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: FlushMRSOp(a66b932d884e4df8a78841103d88ded0) complete. Timing: real 0.025s user 0.025s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":212,"dirs.queue_time_us":244,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":1042,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":466,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":298,"threads_started":3,"wal-append.queue_time_us":210}
I20240502 02:01:18.260098 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling UndoDeltaBlockGCOp(a66b932d884e4df8a78841103d88ded0): 4834 bytes on disk
I20240502 02:01:18.260833 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: UndoDeltaBlockGCOp(a66b932d884e4df8a78841103d88ded0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240502 02:01:18.322484 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling FlushMRSOp(b69d6e907b3b400f93721981732c006b): perf score=0.033366
I20240502 02:01:18.330543 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling FlushMRSOp(dcc5daef41bc44c4855f2bcb44e1c6d5): perf score=0.033372
I20240502 02:01:18.359328 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: FlushMRSOp(b69d6e907b3b400f93721981732c006b) complete. Timing: real 0.036s user 0.033s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":663,"dirs.queue_time_us":8287,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":956,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":664,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1097,"threads_started":3,"wal-append.queue_time_us":1558}
I20240502 02:01:18.361065 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling UndoDeltaBlockGCOp(b69d6e907b3b400f93721981732c006b): 4581 bytes on disk
I20240502 02:01:18.368214 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: UndoDeltaBlockGCOp(b69d6e907b3b400f93721981732c006b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240502 02:01:18.370918 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling FlushMRSOp(b69d6e907b3b400f93721981732c006b): perf score=0.033381
I20240502 02:01:18.433454 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: FlushMRSOp(b69d6e907b3b400f93721981732c006b) complete. Timing: real 0.062s user 0.029s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":11492,"dirs.queue_time_us":12938,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":2757,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":564,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":304,"threads_started":3,"wal-append.queue_time_us":5388}
I20240502 02:01:18.437335 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: FlushMRSOp(dcc5daef41bc44c4855f2bcb44e1c6d5) complete. Timing: real 0.106s user 0.035s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":208,"dirs.queue_time_us":6286,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1470,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":636,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":314,"threads_started":3,"wal-append.queue_time_us":8228}
I20240502 02:01:18.439097 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling UndoDeltaBlockGCOp(dcc5daef41bc44c4855f2bcb44e1c6d5): 4921 bytes on disk
I20240502 02:01:18.439744 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: UndoDeltaBlockGCOp(dcc5daef41bc44c4855f2bcb44e1c6d5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240502 02:01:18.442431 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling UndoDeltaBlockGCOp(b69d6e907b3b400f93721981732c006b): 4581 bytes on disk
I20240502 02:01:18.444213 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: UndoDeltaBlockGCOp(b69d6e907b3b400f93721981732c006b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240502 02:01:18.567842 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:18.573592 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:18.575896 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:18.594470 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:18.612164 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:18.630594 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:18.664151 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:18.669860 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:18.767161 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:01:18.941835 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling FlushMRSOp(0359dfbfd11746f1bf62a465f14634ec): perf score=0.033344
I20240502 02:01:18.946146 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling FlushMRSOp(0359dfbfd11746f1bf62a465f14634ec): perf score=0.033343
W20240502 02:01:18.951723 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:01:18.964340 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: FlushMRSOp(0359dfbfd11746f1bf62a465f14634ec) complete. Timing: real 0.022s user 0.019s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":211,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":2042,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":673,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":768,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":156}
I20240502 02:01:18.965377 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling UndoDeltaBlockGCOp(0359dfbfd11746f1bf62a465f14634ec): 4834 bytes on disk
W20240502 02:01:18.965783 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:01:18.966174 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: UndoDeltaBlockGCOp(0359dfbfd11746f1bf62a465f14634ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240502 02:01:18.970259 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: FlushMRSOp(0359dfbfd11746f1bf62a465f14634ec) complete. Timing: real 0.024s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6739,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":2161,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":589,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":155,"threads_started":2,"wal-append.queue_time_us":5038}
I20240502 02:01:18.971570 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling UndoDeltaBlockGCOp(0359dfbfd11746f1bf62a465f14634ec): 4834 bytes on disk
I20240502 02:01:18.973014 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: UndoDeltaBlockGCOp(0359dfbfd11746f1bf62a465f14634ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 02:01:18.982473 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:18.982774 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:18.984335 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:18.991124 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:18.992257 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:19.000627 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:19.014067 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling FlushMRSOp(c8c237c50773494abf8813494be0f184): perf score=0.033364
I20240502 02:01:19.015677 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:19.035126 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: FlushMRSOp(c8c237c50773494abf8813494be0f184) complete. Timing: real 0.017s user 0.011s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3239,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1497,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":633,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":634,"threads_started":2,"wal-append.queue_time_us":185}
I20240502 02:01:19.036072 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling UndoDeltaBlockGCOp(c8c237c50773494abf8813494be0f184): 4921 bytes on disk
I20240502 02:01:19.036518 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: UndoDeltaBlockGCOp(c8c237c50773494abf8813494be0f184) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240502 02:01:19.045764 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:19.121459 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling FlushMRSOp(47a276e9780e448a9b3c6eab67925514): perf score=0.033392
I20240502 02:01:19.133826 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: FlushMRSOp(47a276e9780e448a9b3c6eab67925514) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":223,"dirs.run_cpu_time_us":286,"dirs.run_wall_time_us":1437,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":498,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":172,"threads_started":2,"wal-append.queue_time_us":169}
I20240502 02:01:19.134701 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling UndoDeltaBlockGCOp(47a276e9780e448a9b3c6eab67925514): 4581 bytes on disk
I20240502 02:01:19.135056 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: UndoDeltaBlockGCOp(47a276e9780e448a9b3c6eab67925514) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
W20240502 02:01:19.235080 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:01:19.237179 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:01:19.288053 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling FlushMRSOp(8529a93abd6e4d62bbc0bddcf2fda78a): perf score=0.033339
I20240502 02:01:19.308488 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: FlushMRSOp(8529a93abd6e4d62bbc0bddcf2fda78a) complete. Timing: real 0.020s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1463,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":504,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":79,"threads_started":1,"wal-append.queue_time_us":216}
I20240502 02:01:19.309988 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling UndoDeltaBlockGCOp(8529a93abd6e4d62bbc0bddcf2fda78a): 4834 bytes on disk
I20240502 02:01:19.311720 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: UndoDeltaBlockGCOp(8529a93abd6e4d62bbc0bddcf2fda78a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240502 02:01:19.330263 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:01:19.333971 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:19.339071 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:19.341143 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:19.341998 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:19.349256 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:01:19.351684 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:19.352659 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:19.353909 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:19.386438 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling FlushMRSOp(17235e019e644b258266636fb6be6648): perf score=0.033368
I20240502 02:01:19.401257 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: FlushMRSOp(17235e019e644b258266636fb6be6648) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1610,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":649,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":156}
I20240502 02:01:19.402385 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling UndoDeltaBlockGCOp(17235e019e644b258266636fb6be6648): 4581 bytes on disk
I20240502 02:01:19.402915 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: UndoDeltaBlockGCOp(17235e019e644b258266636fb6be6648) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 02:01:19.468386 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling FlushMRSOp(9ecbf1c8ee8c46b195d2374a5db81ac0): perf score=0.033393
I20240502 02:01:19.475131 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling FlushMRSOp(8529a93abd6e4d62bbc0bddcf2fda78a): perf score=0.033391
I20240502 02:01:19.481449 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: FlushMRSOp(9ecbf1c8ee8c46b195d2374a5db81ac0) complete. Timing: real 0.013s user 0.009s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":211,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":1672,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":456,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":1792,"thread_start_us":171,"threads_started":2,"wal-append.queue_time_us":159}
I20240502 02:01:19.482265 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling UndoDeltaBlockGCOp(9ecbf1c8ee8c46b195d2374a5db81ac0): 4921 bytes on disk
I20240502 02:01:19.482661 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: UndoDeltaBlockGCOp(9ecbf1c8ee8c46b195d2374a5db81ac0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240502 02:01:19.488512 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: FlushMRSOp(8529a93abd6e4d62bbc0bddcf2fda78a) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":187,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":1413,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":491,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":155,"threads_started":2,"wal-append.queue_time_us":136}
I20240502 02:01:19.489212 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling UndoDeltaBlockGCOp(8529a93abd6e4d62bbc0bddcf2fda78a): 4834 bytes on disk
I20240502 02:01:19.489614 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: UndoDeltaBlockGCOp(8529a93abd6e4d62bbc0bddcf2fda78a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
W20240502 02:01:19.523485 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:19.686610 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:19.691263 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:19.694859 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:19.703096 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:19.705397 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:19.716475 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:19.717031 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:19.767594 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:01:19.976660 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:01:19.984608 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling FlushMRSOp(9f6c07e3923d47f29ef32561d7f438e1): perf score=0.033337
I20240502 02:01:20.000327 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: FlushMRSOp(9f6c07e3923d47f29ef32561d7f438e1) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":337,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":3360,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":563,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":587}
I20240502 02:01:20.001165 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling UndoDeltaBlockGCOp(9f6c07e3923d47f29ef32561d7f438e1): 4921 bytes on disk
I20240502 02:01:20.001605 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: UndoDeltaBlockGCOp(9f6c07e3923d47f29ef32561d7f438e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240502 02:01:20.024034 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:20.024374 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:20.028687 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:20.032733 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:20.035077 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:20.040684 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:20.051374 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:20.055380 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:20.155829 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling FlushMRSOp(d4d429fe60e74108bc0c9ca8509365f2): perf score=0.033385
I20240502 02:01:20.168531 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: FlushMRSOp(d4d429fe60e74108bc0c9ca8509365f2) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":184,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1552,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":434,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":148}
I20240502 02:01:20.169356 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling UndoDeltaBlockGCOp(d4d429fe60e74108bc0c9ca8509365f2): 4581 bytes on disk
I20240502 02:01:20.169759 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: UndoDeltaBlockGCOp(d4d429fe60e74108bc0c9ca8509365f2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
W20240502 02:01:20.189644 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:01:20.361539 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:20.366030 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:20.366569 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:20.371740 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:20.380280 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:20.383704 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:20.404997 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:20.492790 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling FlushMRSOp(e0e05f96dbc542eb999c16dcc0620666): perf score=0.033339
I20240502 02:01:20.503455 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling FlushMRSOp(818dbc9a8e054eeaa122e7919a311806): perf score=0.033340
I20240502 02:01:20.504595 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: FlushMRSOp(e0e05f96dbc542eb999c16dcc0620666) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":246,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1534,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":173,"threads_started":2,"wal-append.queue_time_us":152}
I20240502 02:01:20.505227 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling UndoDeltaBlockGCOp(e0e05f96dbc542eb999c16dcc0620666): 4582 bytes on disk
I20240502 02:01:20.505586 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: UndoDeltaBlockGCOp(e0e05f96dbc542eb999c16dcc0620666) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240502 02:01:20.516672 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: FlushMRSOp(818dbc9a8e054eeaa122e7919a311806) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":161,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":1558,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":155,"threads_started":2,"wal-append.queue_time_us":156}
I20240502 02:01:20.517416 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling UndoDeltaBlockGCOp(818dbc9a8e054eeaa122e7919a311806): 4834 bytes on disk
I20240502 02:01:20.517762 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: UndoDeltaBlockGCOp(818dbc9a8e054eeaa122e7919a311806) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240502 02:01:20.565996 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling FlushMRSOp(44c55b199a24432e902cce9121fc0780): perf score=0.033358
I20240502 02:01:20.578523 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: FlushMRSOp(44c55b199a24432e902cce9121fc0780) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":213,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":1600,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":432,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":186,"threads_started":2,"wal-append.queue_time_us":161}
I20240502 02:01:20.579298 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling UndoDeltaBlockGCOp(44c55b199a24432e902cce9121fc0780): 4920 bytes on disk
I20240502 02:01:20.579684 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: UndoDeltaBlockGCOp(44c55b199a24432e902cce9121fc0780) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
W20240502 02:01:20.611460 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:20.658438 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 02:01:20.671871 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling FlushMRSOp(e0e05f96dbc542eb999c16dcc0620666): perf score=0.033386
I20240502 02:01:20.687714 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: FlushMRSOp(e0e05f96dbc542eb999c16dcc0620666) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":193,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1610,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":573,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":164,"threads_started":2,"wal-append.queue_time_us":151}
I20240502 02:01:20.688601 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling UndoDeltaBlockGCOp(e0e05f96dbc542eb999c16dcc0620666): 4582 bytes on disk
I20240502 02:01:20.688977 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: UndoDeltaBlockGCOp(e0e05f96dbc542eb999c16dcc0620666) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 02:01:20.701488 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:20.702265 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:20.705437 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:20.705860 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:20.713994 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:20.719537 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:20.725528 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:01:20.739651 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:20.779891 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:01:20.940362 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling FlushMRSOp(48eeb721a60c4be888041bd31cfeab7a): perf score=0.033344
I20240502 02:01:20.954293 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: FlushMRSOp(48eeb721a60c4be888041bd31cfeab7a) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1534,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":485,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":114,"threads_started":1,"wal-append.queue_time_us":224}
I20240502 02:01:20.955144 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling UndoDeltaBlockGCOp(48eeb721a60c4be888041bd31cfeab7a): 4921 bytes on disk
I20240502 02:01:20.955560 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: UndoDeltaBlockGCOp(48eeb721a60c4be888041bd31cfeab7a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 02:01:21.019470 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling FlushMRSOp(39d9e48f67f34ebe850fc388ecd94ca3): perf score=0.033367
I20240502 02:01:21.028666 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:21.031109 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:21.037505 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:21.039232 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:21.042457 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:21.058259 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:21.063567 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:21.066649 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:21.087134 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling FlushMRSOp(4154be3f1f3c428580463c419d69892e): perf score=0.033384
I20240502 02:01:21.093963 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: FlushMRSOp(39d9e48f67f34ebe850fc388ecd94ca3) complete. Timing: real 0.074s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":208,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1680,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7792,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":168,"threads_started":2,"wal-append.queue_time_us":171}
I20240502 02:01:21.094869 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling UndoDeltaBlockGCOp(39d9e48f67f34ebe850fc388ecd94ca3): 4834 bytes on disk
I20240502 02:01:21.095288 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: UndoDeltaBlockGCOp(39d9e48f67f34ebe850fc388ecd94ca3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240502 02:01:21.101967 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: FlushMRSOp(4154be3f1f3c428580463c419d69892e) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":185,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1529,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":463,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":129}
I20240502 02:01:21.103212 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling UndoDeltaBlockGCOp(4154be3f1f3c428580463c419d69892e): 4581 bytes on disk
I20240502 02:01:21.103770 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: UndoDeltaBlockGCOp(4154be3f1f3c428580463c419d69892e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
W20240502 02:01:21.257053 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:01:21.302318 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:01:21.387327 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:21.387682 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:21.393982 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:21.394554 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:21.394807 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:21.399780 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:21.406574 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:21.447579 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:21.457477 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling FlushMRSOp(9313079c48c94ab1b7e022a91befafcd): perf score=0.033355
I20240502 02:01:21.470124 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: FlushMRSOp(9313079c48c94ab1b7e022a91befafcd) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":194,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":1641,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":510,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":155,"threads_started":2,"wal-append.queue_time_us":145}
I20240502 02:01:21.470996 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling UndoDeltaBlockGCOp(9313079c48c94ab1b7e022a91befafcd): 4834 bytes on disk
I20240502 02:01:21.471402 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: UndoDeltaBlockGCOp(9313079c48c94ab1b7e022a91befafcd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240502 02:01:21.515103 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:01:21.540813 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:01:21.597298 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling FlushMRSOp(ed7b67af7b1c4255a7c2377570b58e70): perf score=0.033394
I20240502 02:01:21.606048 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling FlushMRSOp(ed7b67af7b1c4255a7c2377570b58e70): perf score=0.033397
I20240502 02:01:21.611830 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: FlushMRSOp(ed7b67af7b1c4255a7c2377570b58e70) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":259,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1823,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":713,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":200,"threads_started":2,"wal-append.queue_time_us":265}
I20240502 02:01:21.612704 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling UndoDeltaBlockGCOp(ed7b67af7b1c4255a7c2377570b58e70): 4581 bytes on disk
I20240502 02:01:21.613119 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: UndoDeltaBlockGCOp(ed7b67af7b1c4255a7c2377570b58e70) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240502 02:01:21.622871 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: FlushMRSOp(ed7b67af7b1c4255a7c2377570b58e70) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":230,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1452,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":463,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":178,"threads_started":2,"wal-append.queue_time_us":164}
I20240502 02:01:21.623812 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling UndoDeltaBlockGCOp(ed7b67af7b1c4255a7c2377570b58e70): 4581 bytes on disk
I20240502 02:01:21.624709 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: UndoDeltaBlockGCOp(ed7b67af7b1c4255a7c2377570b58e70) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240502 02:01:21.663916 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:01:21.716907 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:21.717572 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:21.718523 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:21.728441 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:21.732051 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:21.748438 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:21.752490 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:21.784701 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:01:21.878449 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:01:21.891366 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:01:21.949255 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:21.977692 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling FlushMRSOp(7345f51bd6f84b3880838c2460bafb75): perf score=0.033360
I20240502 02:01:21.994293 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: FlushMRSOp(7345f51bd6f84b3880838c2460bafb75) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1771,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":510,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":174,"threads_started":2,"wal-append.queue_time_us":187}
I20240502 02:01:21.995227 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling UndoDeltaBlockGCOp(7345f51bd6f84b3880838c2460bafb75): 4834 bytes on disk
I20240502 02:01:21.995663 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: UndoDeltaBlockGCOp(7345f51bd6f84b3880838c2460bafb75) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 02:01:22.010030 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling FlushMRSOp(a4fb9af11abc49d496b5c3d8048c22c8): perf score=0.033372
I20240502 02:01:22.026597 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: FlushMRSOp(a4fb9af11abc49d496b5c3d8048c22c8) complete. Timing: real 0.016s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":183,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":1658,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":522,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":171,"threads_started":2,"wal-append.queue_time_us":176}
I20240502 02:01:22.027457 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling UndoDeltaBlockGCOp(a4fb9af11abc49d496b5c3d8048c22c8): 4921 bytes on disk
I20240502 02:01:22.027891 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: UndoDeltaBlockGCOp(a4fb9af11abc49d496b5c3d8048c22c8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 02:01:22.056114 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:22.057883 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:22.059458 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:22.066761 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:22.070991 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:22.079666 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:22.082132 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:22.126732 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling FlushMRSOp(7fa358140eb742b587a4d6a203738b4f): perf score=0.033402
I20240502 02:01:22.138725 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: FlushMRSOp(7fa358140eb742b587a4d6a203738b4f) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":216,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":1532,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":452,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":170}
I20240502 02:01:22.139537 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling UndoDeltaBlockGCOp(7fa358140eb742b587a4d6a203738b4f): 4581 bytes on disk
I20240502 02:01:22.140012 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: UndoDeltaBlockGCOp(7fa358140eb742b587a4d6a203738b4f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240502 02:01:22.250802 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:01:22.396672 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:01:22.398346 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:22.399861 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:22.400023 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:22.401912 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:22.402390 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:22.407883 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:22.408723 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:22.433889 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:22.497660 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling FlushMRSOp(a0548274c47148d9b611fd3c978a0c34): perf score=0.033361
I20240502 02:01:22.508997 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: FlushMRSOp(a0548274c47148d9b611fd3c978a0c34) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":234,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":1844,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":457,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":162}
I20240502 02:01:22.509815 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling FlushMRSOp(14e3c78c65334b52aca79d67c4bf45a2): perf score=0.033363
I20240502 02:01:22.521442 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: FlushMRSOp(14e3c78c65334b52aca79d67c4bf45a2) complete. Timing: real 0.011s user 0.006s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":73,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":1575,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":447,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":87,"threads_started":1,"wal-append.queue_time_us":165}
I20240502 02:01:22.522541 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling UndoDeltaBlockGCOp(14e3c78c65334b52aca79d67c4bf45a2): 4834 bytes on disk
I20240502 02:01:22.523010 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: UndoDeltaBlockGCOp(14e3c78c65334b52aca79d67c4bf45a2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240502 02:01:22.523664 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling UndoDeltaBlockGCOp(a0548274c47148d9b611fd3c978a0c34): 4581 bytes on disk
I20240502 02:01:22.524071 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: UndoDeltaBlockGCOp(a0548274c47148d9b611fd3c978a0c34) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240502 02:01:22.579154 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:22.579824 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 02:01:22.616037 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling FlushMRSOp(1c54b912f802461c8cac092ac3222c87): perf score=0.033393
I20240502 02:01:22.632000 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: FlushMRSOp(1c54b912f802461c8cac092ac3222c87) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":200,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1646,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":510,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":178,"threads_started":2,"wal-append.queue_time_us":168}
I20240502 02:01:22.632740 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling UndoDeltaBlockGCOp(1c54b912f802461c8cac092ac3222c87): 4921 bytes on disk
I20240502 02:01:22.633152 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: UndoDeltaBlockGCOp(1c54b912f802461c8cac092ac3222c87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 02:01:22.642097 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling FlushMRSOp(1c54b912f802461c8cac092ac3222c87): perf score=0.033402
I20240502 02:01:22.654801 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: FlushMRSOp(1c54b912f802461c8cac092ac3222c87) complete. Timing: real 0.012s user 0.011s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":198,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1399,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":480,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":172,"threads_started":2,"wal-append.queue_time_us":154}
I20240502 02:01:22.655552 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling UndoDeltaBlockGCOp(1c54b912f802461c8cac092ac3222c87): 4921 bytes on disk
I20240502 02:01:22.655908 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: UndoDeltaBlockGCOp(1c54b912f802461c8cac092ac3222c87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240502 02:01:22.729406 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:22.732290 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:22.732700 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:22.734632 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:22.746008 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:22.746556 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:22.748181 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:22.780793 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:01:22.928169 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:22.947851 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:01:23.072973 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:23.075510 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:23.077188 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:23.084439 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:23.085651 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:23.089635 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:23.098884 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:23.134828 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling FlushMRSOp(347b161e9ec84640a0668836ce9e29eb): perf score=0.033355
I20240502 02:01:23.146859 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: FlushMRSOp(347b161e9ec84640a0668836ce9e29eb) complete. Timing: real 0.012s user 0.010s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":179,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1583,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":481,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":167,"threads_started":2,"wal-append.queue_time_us":158}
I20240502 02:01:23.147521 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling UndoDeltaBlockGCOp(347b161e9ec84640a0668836ce9e29eb): 4835 bytes on disk
I20240502 02:01:23.147881 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: UndoDeltaBlockGCOp(347b161e9ec84640a0668836ce9e29eb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240502 02:01:23.158084 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling FlushMRSOp(347b161e9ec84640a0668836ce9e29eb): perf score=0.033363
I20240502 02:01:23.170907 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: FlushMRSOp(347b161e9ec84640a0668836ce9e29eb) complete. Timing: real 0.013s user 0.010s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":168,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1595,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":478,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":145}
I20240502 02:01:23.171733 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling UndoDeltaBlockGCOp(347b161e9ec84640a0668836ce9e29eb): 4835 bytes on disk
I20240502 02:01:23.172132 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: UndoDeltaBlockGCOp(347b161e9ec84640a0668836ce9e29eb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240502 02:01:23.276796 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling FlushMRSOp(91db5c1b3d2f4d22b5882a61ea0395c7): perf score=0.033398
I20240502 02:01:23.281668 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling FlushMRSOp(4e44267bd266407f9b72c66e3d3579bb): perf score=0.033394
I20240502 02:01:23.291713 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: FlushMRSOp(91db5c1b3d2f4d22b5882a61ea0395c7) complete. Timing: real 0.015s user 0.009s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":241,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1737,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":544,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":225,"threads_started":2,"wal-append.queue_time_us":173}
I20240502 02:01:23.292717 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling FlushMRSOp(4e44267bd266407f9b72c66e3d3579bb): perf score=0.033399
I20240502 02:01:23.297477 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: FlushMRSOp(4e44267bd266407f9b72c66e3d3579bb) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":173,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1274,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":555,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":139,"threads_started":2,"wal-append.queue_time_us":141}
I20240502 02:01:23.298117 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling UndoDeltaBlockGCOp(4e44267bd266407f9b72c66e3d3579bb): 4581 bytes on disk
I20240502 02:01:23.298570 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: UndoDeltaBlockGCOp(4e44267bd266407f9b72c66e3d3579bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240502 02:01:23.305223 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: FlushMRSOp(4e44267bd266407f9b72c66e3d3579bb) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1110,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":499,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":102,"threads_started":1,"wal-append.queue_time_us":184}
I20240502 02:01:23.305939 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling UndoDeltaBlockGCOp(91db5c1b3d2f4d22b5882a61ea0395c7): 4921 bytes on disk
I20240502 02:01:23.306279 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: UndoDeltaBlockGCOp(91db5c1b3d2f4d22b5882a61ea0395c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240502 02:01:23.306890 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling UndoDeltaBlockGCOp(4e44267bd266407f9b72c66e3d3579bb): 4581 bytes on disk
I20240502 02:01:23.307155 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: UndoDeltaBlockGCOp(4e44267bd266407f9b72c66e3d3579bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240502 02:01:23.323524 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:23.407745 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:23.409077 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:23.409425 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:23.418076 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:23.418386 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:23.429435 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:23.431773 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:23.500857 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:01:23.601769 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:01:23.626935 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 02:01:23.740118 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:23.741974 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:23.743624 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:23.752137 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:23.757452 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:23.762238 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:23.771324 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:23.793128 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:23.862268 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:01:24.050729 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 02:01:24.076319 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:24.078709 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:24.082849 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:24.089139 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:24.102381 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:24.103178 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:24.114230 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:24.207397 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:01:24.221274 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:01:24.314246 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:01:24.367278 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:24.410444 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:24.410794 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:24.411866 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:24.421841 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:24.422055 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:24.430910 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:24.453706 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:24.737696 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:24.741768 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:24.745087 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:24.746567 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:24.746603 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:24.746886 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:24.757918 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:24.925628 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:24.949129 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:01:25.038439 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:01:25.076201 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:25.076725 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:25.077035 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:25.093565 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:25.097769 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:25.107082 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:25.119504 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:25.338502 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:25.419016 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:25.419368 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:25.420775 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:25.428084 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:25.431528 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:25.431972 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:25.441466 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:25.459676 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:01:25.602982 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 02:01:25.750456 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:25.753769 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:25.759814 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:25.765408 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:25.773406 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:25.775686 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:25.784247 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:25.918002 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 02:01:25.957348 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:25.967145 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:01:26.081848 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:26.086846 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:26.087244 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:26.087877 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:26.090590 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:26.103492 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:26.115623 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:26.139937 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:01:26.265900 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:01:26.273180 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:01:26.409425 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:26.418702 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:26.418869 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:26.419082 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:01:26.419150 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:26.443449 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:26.447860 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:26.448813 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:26.553942 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:26.680408 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:01:26.740458 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:26.742761 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:01:26.744709 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:26.751834 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:26.753185 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:26.763540 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:26.764060 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:01:26.766139 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:26.774029 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:27.070317 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:27.079083 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:27.079838 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:27.079906 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:27.082062 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:27.082571 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:27.104878 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 02:01:27.105917 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:27.156019 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:27.184768 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling FlushMRSOp(4154be3f1f3c428580463c419d69892e): perf score=0.033344
I20240502 02:01:27.210505 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: FlushMRSOp(4154be3f1f3c428580463c419d69892e) complete. Timing: real 0.025s user 0.025s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":216,"dirs.queue_time_us":189,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1172,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":395,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":285,"threads_started":3,"wal-append.queue_time_us":166}
I20240502 02:01:27.211644 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling FlushMRSOp(347b161e9ec84640a0668836ce9e29eb): perf score=0.033351
I20240502 02:01:27.232236 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: FlushMRSOp(347b161e9ec84640a0668836ce9e29eb) complete. Timing: real 0.020s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":70,"dirs.run_cpu_time_us":297,"dirs.run_wall_time_us":1696,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":539,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":123,"threads_started":1,"wal-append.queue_time_us":220}
I20240502 02:01:27.233062 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling UndoDeltaBlockGCOp(347b161e9ec84640a0668836ce9e29eb): 4835 bytes on disk
I20240502 02:01:27.233729 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: UndoDeltaBlockGCOp(347b161e9ec84640a0668836ce9e29eb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 02:01:27.234447 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling UndoDeltaBlockGCOp(4154be3f1f3c428580463c419d69892e): 4581 bytes on disk
I20240502 02:01:27.234792 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: UndoDeltaBlockGCOp(4154be3f1f3c428580463c419d69892e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240502 02:01:27.335718 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:01:27.356748 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling FlushMRSOp(14e3c78c65334b52aca79d67c4bf45a2): perf score=0.033374
I20240502 02:01:27.395023 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling FlushMRSOp(9ecbf1c8ee8c46b195d2374a5db81ac0): perf score=0.033401
I20240502 02:01:27.409273 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: FlushMRSOp(14e3c78c65334b52aca79d67c4bf45a2) complete. Timing: real 0.052s user 0.027s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":5509,"dirs.queue_time_us":8089,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":575,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":258,"threads_started":3,"wal-append.queue_time_us":338}
I20240502 02:01:27.410841 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling UndoDeltaBlockGCOp(14e3c78c65334b52aca79d67c4bf45a2): 4834 bytes on disk
I20240502 02:01:27.411477 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: UndoDeltaBlockGCOp(14e3c78c65334b52aca79d67c4bf45a2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240502 02:01:27.412216 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling FlushMRSOp(17235e019e644b258266636fb6be6648): perf score=0.033364
W20240502 02:01:27.435286 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:01:27.467223 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: FlushMRSOp(17235e019e644b258266636fb6be6648) complete. Timing: real 0.049s user 0.014s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":217,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1564,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":97,"threads_started":1,"wal-append.queue_time_us":168}
I20240502 02:01:27.468451 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling FlushMRSOp(034cf1c3d47a4f7785c16d16815c229d): perf score=0.033378
I20240502 02:01:27.486476 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling FlushMRSOp(7345f51bd6f84b3880838c2460bafb75): perf score=0.033378
I20240502 02:01:27.487679 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:27.505728 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:27.521620 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:27.522687 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:27.522943 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:27.523476 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: FlushMRSOp(7345f51bd6f84b3880838c2460bafb75) complete. Timing: real 0.037s user 0.013s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":119,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1514,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":538,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":113,"threads_started":1,"wal-append.queue_time_us":6772}
I20240502 02:01:27.524585 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling UndoDeltaBlockGCOp(7345f51bd6f84b3880838c2460bafb75): 4834 bytes on disk
I20240502 02:01:27.525118 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: UndoDeltaBlockGCOp(7345f51bd6f84b3880838c2460bafb75) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240502 02:01:27.528064 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling FlushMRSOp(9f6c07e3923d47f29ef32561d7f438e1): perf score=0.033389
I20240502 02:01:27.542634 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:27.566661 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:27.596922 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: FlushMRSOp(9ecbf1c8ee8c46b195d2374a5db81ac0) complete. Timing: real 0.201s user 0.031s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":4197,"dirs.queue_time_us":123019,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":933,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1615,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":20224,"thread_start_us":264,"threads_started":3,"wal-append.queue_time_us":115446}
I20240502 02:01:27.598621 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling FlushMRSOp(17235e019e644b258266636fb6be6648): perf score=0.033449
I20240502 02:01:27.617200 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: FlushMRSOp(9f6c07e3923d47f29ef32561d7f438e1) complete. Timing: real 0.089s user 0.013s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1454,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":504,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":111,"threads_started":1,"wal-append.queue_time_us":35305}
I20240502 02:01:27.618239 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling UndoDeltaBlockGCOp(9f6c07e3923d47f29ef32561d7f438e1): 4921 bytes on disk
I20240502 02:01:27.618808 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: UndoDeltaBlockGCOp(9f6c07e3923d47f29ef32561d7f438e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 02:01:27.619637 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling FlushMRSOp(39d9e48f67f34ebe850fc388ecd94ca3): perf score=0.033414
I20240502 02:01:27.634542 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: FlushMRSOp(034cf1c3d47a4f7785c16d16815c229d) complete. Timing: real 0.161s user 0.016s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":4,"lbm_write_time_us":568,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":102,"threads_started":1,"wal-append.queue_time_us":196}
I20240502 02:01:27.635759 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling FlushMRSOp(818dbc9a8e054eeaa122e7919a311806): perf score=0.033414
I20240502 02:01:27.648897 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: FlushMRSOp(17235e019e644b258266636fb6be6648) complete. Timing: real 0.050s user 0.020s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1439,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":612,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":106,"threads_started":1,"wal-append.queue_time_us":12775}
I20240502 02:01:27.648897 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: FlushMRSOp(39d9e48f67f34ebe850fc388ecd94ca3) complete. Timing: real 0.029s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1501,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7326,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":119,"threads_started":1,"wal-append.queue_time_us":868}
I20240502 02:01:27.649816 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling UndoDeltaBlockGCOp(39d9e48f67f34ebe850fc388ecd94ca3): 4834 bytes on disk
I20240502 02:01:27.649973 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling UndoDeltaBlockGCOp(9ecbf1c8ee8c46b195d2374a5db81ac0): 4921 bytes on disk
I20240502 02:01:27.650275 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: UndoDeltaBlockGCOp(39d9e48f67f34ebe850fc388ecd94ca3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240502 02:01:27.650754 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: UndoDeltaBlockGCOp(9ecbf1c8ee8c46b195d2374a5db81ac0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":155,"lbm_reads_lt_1ms":4}
I20240502 02:01:27.651712 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling FlushMRSOp(7345f51bd6f84b3880838c2460bafb75): perf score=0.033451
I20240502 02:01:27.658690 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: FlushMRSOp(818dbc9a8e054eeaa122e7919a311806) complete. Timing: real 0.023s user 0.016s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":14788,"compiler_manager_pool.run_cpu_time_us":13,"compiler_manager_pool.run_wall_time_us":14,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1199,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":558,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":119,"threads_started":1,"wal-append.queue_time_us":204}
I20240502 02:01:27.659649 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling UndoDeltaBlockGCOp(034cf1c3d47a4f7785c16d16815c229d): 4921 bytes on disk
I20240502 02:01:27.660025 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: UndoDeltaBlockGCOp(034cf1c3d47a4f7785c16d16815c229d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240502 02:01:27.660643 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling UndoDeltaBlockGCOp(818dbc9a8e054eeaa122e7919a311806): 4834 bytes on disk
I20240502 02:01:27.660969 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: UndoDeltaBlockGCOp(818dbc9a8e054eeaa122e7919a311806) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 02:01:27.661496 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling UndoDeltaBlockGCOp(17235e019e644b258266636fb6be6648): 4581 bytes on disk
I20240502 02:01:27.661857 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: UndoDeltaBlockGCOp(17235e019e644b258266636fb6be6648) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240502 02:01:27.669315 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: FlushMRSOp(7345f51bd6f84b3880838c2460bafb75) complete. Timing: real 0.017s user 0.006s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1421,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":402,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":103,"threads_started":1,"wal-append.queue_time_us":189}
I20240502 02:01:27.670291 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling FlushMRSOp(9313079c48c94ab1b7e022a91befafcd): perf score=0.033441
I20240502 02:01:27.684814 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: FlushMRSOp(9313079c48c94ab1b7e022a91befafcd) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1351,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":581,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":105,"threads_started":1,"wal-append.queue_time_us":203}
I20240502 02:01:27.685549 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling UndoDeltaBlockGCOp(7345f51bd6f84b3880838c2460bafb75): 4834 bytes on disk
I20240502 02:01:27.685885 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: UndoDeltaBlockGCOp(7345f51bd6f84b3880838c2460bafb75) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240502 02:01:27.686460 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling UndoDeltaBlockGCOp(9313079c48c94ab1b7e022a91befafcd): 4834 bytes on disk
I20240502 02:01:27.686745 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: UndoDeltaBlockGCOp(9313079c48c94ab1b7e022a91befafcd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4}
I20240502 02:01:27.687314 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling UndoDeltaBlockGCOp(17235e019e644b258266636fb6be6648): 4581 bytes on disk
I20240502 02:01:27.687575 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: UndoDeltaBlockGCOp(17235e019e644b258266636fb6be6648) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4}
W20240502 02:01:27.745452 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:01:27.754108 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:27.754989 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:27.757835 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:01:27.876250 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:01:27.888294 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:27.889015 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:27.889608 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:27.905062 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:27.906555 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:27.908605 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:27.932538 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:28.163796 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling FlushMRSOp(47a276e9780e448a9b3c6eab67925514): perf score=0.033348
I20240502 02:01:28.176206 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: FlushMRSOp(47a276e9780e448a9b3c6eab67925514) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":198,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1522,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":496,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":192,"threads_started":2,"wal-append.queue_time_us":171}
I20240502 02:01:28.176934 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling UndoDeltaBlockGCOp(47a276e9780e448a9b3c6eab67925514): 4581 bytes on disk
I20240502 02:01:28.177285 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: UndoDeltaBlockGCOp(47a276e9780e448a9b3c6eab67925514) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240502 02:01:28.177901 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling FlushMRSOp(91db5c1b3d2f4d22b5882a61ea0395c7): perf score=0.033349
I20240502 02:01:28.191812 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: FlushMRSOp(91db5c1b3d2f4d22b5882a61ea0395c7) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1422,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":416,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":82,"threads_started":1,"wal-append.queue_time_us":174}
I20240502 02:01:28.192900 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling FlushMRSOp(a66b932d884e4df8a78841103d88ded0): perf score=0.033349
I20240502 02:01:28.217329 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: FlushMRSOp(a66b932d884e4df8a78841103d88ded0) complete. Timing: real 0.024s user 0.012s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1561,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":688,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":119,"threads_started":1,"wal-append.queue_time_us":896}
I20240502 02:01:28.218307 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling UndoDeltaBlockGCOp(91db5c1b3d2f4d22b5882a61ea0395c7): 4921 bytes on disk
I20240502 02:01:28.218746 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: UndoDeltaBlockGCOp(91db5c1b3d2f4d22b5882a61ea0395c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 02:01:28.219448 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling UndoDeltaBlockGCOp(a66b932d884e4df8a78841103d88ded0): 4834 bytes on disk
I20240502 02:01:28.220019 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: UndoDeltaBlockGCOp(a66b932d884e4df8a78841103d88ded0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240502 02:01:28.232287 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:28.232350 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:28.234874 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:28.240603 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:28.244715 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:28.248219 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:28.262001 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:28.266809 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:28.440227 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling FlushMRSOp(e6ac26fa6de44f7ca68e6c0b6b959db7): perf score=0.033354
I20240502 02:01:28.457509 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: FlushMRSOp(e6ac26fa6de44f7ca68e6c0b6b959db7) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":210,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1715,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":600,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":175,"threads_started":2,"wal-append.queue_time_us":183}
I20240502 02:01:28.458498 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling UndoDeltaBlockGCOp(e6ac26fa6de44f7ca68e6c0b6b959db7): 4835 bytes on disk
I20240502 02:01:28.458920 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: UndoDeltaBlockGCOp(e6ac26fa6de44f7ca68e6c0b6b959db7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240502 02:01:28.471479 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling FlushMRSOp(48eeb721a60c4be888041bd31cfeab7a): perf score=0.033353
I20240502 02:01:28.484819 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: FlushMRSOp(48eeb721a60c4be888041bd31cfeab7a) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":1593,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":449,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":74,"threads_started":1,"wal-append.queue_time_us":152}
I20240502 02:01:28.485616 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling FlushMRSOp(c8c237c50773494abf8813494be0f184): perf score=0.033343
I20240502 02:01:28.499320 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: FlushMRSOp(c8c237c50773494abf8813494be0f184) complete. Timing: real 0.014s user 0.009s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":1486,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":516,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":71,"threads_started":1,"wal-append.queue_time_us":134}
I20240502 02:01:28.500135 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling UndoDeltaBlockGCOp(48eeb721a60c4be888041bd31cfeab7a): 4921 bytes on disk
I20240502 02:01:28.500509 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: UndoDeltaBlockGCOp(48eeb721a60c4be888041bd31cfeab7a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240502 02:01:28.501271 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling UndoDeltaBlockGCOp(c8c237c50773494abf8813494be0f184): 4921 bytes on disk
I20240502 02:01:28.501612 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: UndoDeltaBlockGCOp(c8c237c50773494abf8813494be0f184) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240502 02:01:28.561626 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:28.561848 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:28.564158 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:28.566915 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:28.578366 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:01:28.578814 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:28.581043 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:28.601979 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:28.655177 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling FlushMRSOp(44c55b199a24432e902cce9121fc0780): perf score=0.033401
I20240502 02:01:28.668206 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: FlushMRSOp(44c55b199a24432e902cce9121fc0780) complete. Timing: real 0.013s user 0.009s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":156,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":1588,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":409,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":157}
I20240502 02:01:28.668913 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling UndoDeltaBlockGCOp(44c55b199a24432e902cce9121fc0780): 4920 bytes on disk
I20240502 02:01:28.669272 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: UndoDeltaBlockGCOp(44c55b199a24432e902cce9121fc0780) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
W20240502 02:01:28.683008 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:01:28.710470 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling FlushMRSOp(48eeb721a60c4be888041bd31cfeab7a): perf score=0.033393
I20240502 02:01:28.723031 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: FlushMRSOp(48eeb721a60c4be888041bd31cfeab7a) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1579,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":439,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":150}
I20240502 02:01:28.723793 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling UndoDeltaBlockGCOp(48eeb721a60c4be888041bd31cfeab7a): 4921 bytes on disk
I20240502 02:01:28.724180 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: UndoDeltaBlockGCOp(48eeb721a60c4be888041bd31cfeab7a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240502 02:01:28.724714 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling FlushMRSOp(1c54b912f802461c8cac092ac3222c87): perf score=0.033378
I20240502 02:01:28.736078 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: FlushMRSOp(1c54b912f802461c8cac092ac3222c87) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":1476,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":376,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":68,"threads_started":1,"wal-append.queue_time_us":139}
I20240502 02:01:28.736807 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling FlushMRSOp(4e44267bd266407f9b72c66e3d3579bb): perf score=0.033381
I20240502 02:01:28.747817 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: FlushMRSOp(4e44267bd266407f9b72c66e3d3579bb) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1340,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":438,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":88,"threads_started":1,"wal-append.queue_time_us":195}
I20240502 02:01:28.748556 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling UndoDeltaBlockGCOp(1c54b912f802461c8cac092ac3222c87): 4921 bytes on disk
I20240502 02:01:28.748903 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: UndoDeltaBlockGCOp(1c54b912f802461c8cac092ac3222c87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240502 02:01:28.749457 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling UndoDeltaBlockGCOp(4e44267bd266407f9b72c66e3d3579bb): 4581 bytes on disk
I20240502 02:01:28.749745 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: UndoDeltaBlockGCOp(4e44267bd266407f9b72c66e3d3579bb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
W20240502 02:01:28.760653 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:28.800261 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:01:28.860617 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:01:28.892694 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:28.894259 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:28.895013 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:28.895002 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:28.903084 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:28.912775 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:28.925483 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:28.959688 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:01:28.996555 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:01:29.130203 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:01:29.171324 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling FlushMRSOp(7fa358140eb742b587a4d6a203738b4f): perf score=0.033385
W20240502 02:01:29.178437 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:29.184991 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: FlushMRSOp(7fa358140eb742b587a4d6a203738b4f) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":172,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1445,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":584,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":153,"threads_started":2,"wal-append.queue_time_us":161}
I20240502 02:01:29.185784 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling UndoDeltaBlockGCOp(7fa358140eb742b587a4d6a203738b4f): 4581 bytes on disk
I20240502 02:01:29.186262 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: UndoDeltaBlockGCOp(7fa358140eb742b587a4d6a203738b4f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 02:01:29.218505 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:29.222594 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:29.223290 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:29.225963 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:29.238240 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:29.239737 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:29.240949 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:29.449014 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:01:29.502440 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling FlushMRSOp(b69d6e907b3b400f93721981732c006b): perf score=0.033383
I20240502 02:01:29.517621 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: FlushMRSOp(b69d6e907b3b400f93721981732c006b) complete. Timing: real 0.015s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":1782,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":624,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":134}
I20240502 02:01:29.518474 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling UndoDeltaBlockGCOp(b69d6e907b3b400f93721981732c006b): 4581 bytes on disk
I20240502 02:01:29.518831 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: UndoDeltaBlockGCOp(b69d6e907b3b400f93721981732c006b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240502 02:01:29.520133 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling FlushMRSOp(d4d429fe60e74108bc0c9ca8509365f2): perf score=0.033372
I20240502 02:01:29.541934 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: FlushMRSOp(d4d429fe60e74108bc0c9ca8509365f2) complete. Timing: real 0.022s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1509,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":611,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":97,"threads_started":1,"wal-append.queue_time_us":2439}
I20240502 02:01:29.542801 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling UndoDeltaBlockGCOp(d4d429fe60e74108bc0c9ca8509365f2): 4581 bytes on disk
I20240502 02:01:29.543237 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: UndoDeltaBlockGCOp(d4d429fe60e74108bc0c9ca8509365f2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 02:01:29.554849 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:29.556223 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:29.556883 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:29.560578 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:29.562271 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:29.574223 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:29.578680 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:29.693993 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:29.815233 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:01:29.880880 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:29.882078 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:29.882870 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:29.887876 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:29.896543 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:29.902590 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:29.907157 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:30.018700 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:01:30.118527 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:30.141799 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:01:30.202571 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:30.202591 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:30.205359 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:30.205718 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:30.207157 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:30.210575 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:30.211270 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:30.522614 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:30.524178 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:30.528301 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:30.535610 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:30.543725 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:30.545032 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:30.545733 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:30.584410 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:01:30.734318 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:30.845897 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:30.847312 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:30.849871 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:30.850606 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:30.852284 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:30.859742 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:30.867779 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:30.918938 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:01:31.119005 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:01:31.172441 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:01:31.173722 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:31.174158 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:31.178014 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:31.182044 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:31.188248 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:01:31.194713 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:01:31.198482 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:31.198998 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:31.211381 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:31.315235 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:31.420471 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:01:31.443313 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling FlushMRSOp(9ecbf1c8ee8c46b195d2374a5db81ac0): perf score=0.033397
I20240502 02:01:31.458868 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: FlushMRSOp(9ecbf1c8ee8c46b195d2374a5db81ac0) complete. Timing: real 0.015s user 0.010s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":717,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":1524,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":482,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":215,"threads_started":2,"wal-append.queue_time_us":155}
I20240502 02:01:31.459754 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling UndoDeltaBlockGCOp(9ecbf1c8ee8c46b195d2374a5db81ac0): 4921 bytes on disk
I20240502 02:01:31.460182 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: UndoDeltaBlockGCOp(9ecbf1c8ee8c46b195d2374a5db81ac0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240502 02:01:31.510550 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:01:31.511880 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:31.512965 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:31.513303 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:31.515014 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:31.515193 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:31.517053 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:31.531688 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:31.692010 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:31.802148 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:01:31.834926 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:31.839090 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:31.839783 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:31.842156 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:31.850558 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:31.852026 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:31.864158 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:31.864300 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:01:32.098187 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:32.137022 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 02:01:32.161511 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:32.167281 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:32.168222 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:32.174667 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:32.176632 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:32.179165 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:32.190574 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:32.227874 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:01:32.307480 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 02:01:32.495918 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:32.496452 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:32.496542 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:32.498498 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:32.499915 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:32.502017 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:32.530946 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:32.630200 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:32.663069 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:01:32.760795 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:01:32.825345 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:32.828445 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:32.834578 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:32.834699 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:32.835765 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:32.838028 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:32.846565 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:32.878121 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:33.018141 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:33.066589 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:01:33.151095 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:33.158092 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:33.160578 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:33.161398 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:33.173697 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:33.176836 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:33.197201 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:33.487219 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:33.490424 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:33.496071 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:33.497064 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:33.507130 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:33.508443 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:33.510538 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:33.544224 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:01:33.546772 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:01:33.607932 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:01:33.632596 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:33.813522 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:33.815717 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:33.821962 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:33.822314 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:33.822750 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:33.836138 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:33.845791 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:33.959592 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:01:33.962208 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:01:34.040117 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:01:34.116299 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:34.139276 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:34.141120 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:34.144879 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:34.145699 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:34.155823 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:34.172274 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:34.182626 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:34.213776 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 02:01:34.474839 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:34.476903 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:34.480159 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:34.482889 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:34.484323 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:34.486136 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:34.507623 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:34.553361 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:01:34.565670 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:01:34.690583 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:01:34.692688 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:34.804122 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:34.805258 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:34.808228 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:34.809618 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:34.821556 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:34.828712 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:34.835083 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:35.133533 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:35.136024 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:35.136775 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:35.145987 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:35.158150 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:35.159497 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:35.179385 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 02:01:35.180181 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:35.221112 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling FlushMRSOp(c8c237c50773494abf8813494be0f184): perf score=0.033349
W20240502 02:01:35.226570 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:35.233415 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: FlushMRSOp(c8c237c50773494abf8813494be0f184) complete. Timing: real 0.012s user 0.009s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":189,"dirs.run_cpu_time_us":280,"dirs.run_wall_time_us":1493,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":464,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":159,"threads_started":2,"wal-append.queue_time_us":167}
I20240502 02:01:35.234155 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling UndoDeltaBlockGCOp(c8c237c50773494abf8813494be0f184): 4921 bytes on disk
I20240502 02:01:35.234557 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: UndoDeltaBlockGCOp(c8c237c50773494abf8813494be0f184) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240502 02:01:35.472764 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:35.480962 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:35.481268 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:35.483073 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:35.483107 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:35.492030 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:35.495918 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:35.625461 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:35.798556 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 02:01:35.803697 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:35.811961 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:35.813148 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:35.815150 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:35.815622 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:35.815842 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:35.836338 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:35.875344 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:01:35.982797 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:01:36.066058 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:36.115557 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:01:36.141276 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:36.142869 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:36.144145 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:36.147991 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:36.150579 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:36.160008 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:36.168964 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:36.195649 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:01:36.275103 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling FlushMRSOp(39d9e48f67f34ebe850fc388ecd94ca3): perf score=0.033382
I20240502 02:01:36.287786 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: FlushMRSOp(39d9e48f67f34ebe850fc388ecd94ca3) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":242,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":1888,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":502,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":175,"threads_started":2,"wal-append.queue_time_us":267}
I20240502 02:01:36.288775 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling UndoDeltaBlockGCOp(39d9e48f67f34ebe850fc388ecd94ca3): 4834 bytes on disk
I20240502 02:01:36.289182 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: UndoDeltaBlockGCOp(39d9e48f67f34ebe850fc388ecd94ca3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240502 02:01:36.389358 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:01:36.434767 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:01:36.471297 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:36.474229 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:36.474578 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:36.475222 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:36.486104 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:01:36.487812 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:36.487975 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:36.508582 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:36.540051 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:36.658327 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:01:36.805094 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:36.805742 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:36.812137 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:36.814091 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:36.814599 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:36.827126 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:36.832183 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:36.989569 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:36.990002 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:01:37.015883 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:01:37.041981 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling FlushMRSOp(a4fb9af11abc49d496b5c3d8048c22c8): perf score=0.033354
I20240502 02:01:37.054683 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: FlushMRSOp(a4fb9af11abc49d496b5c3d8048c22c8) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":173,"dirs.run_cpu_time_us":257,"dirs.run_wall_time_us":1547,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":513,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":149}
I20240502 02:01:37.055477 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling UndoDeltaBlockGCOp(a4fb9af11abc49d496b5c3d8048c22c8): 4921 bytes on disk
I20240502 02:01:37.055917 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: UndoDeltaBlockGCOp(a4fb9af11abc49d496b5c3d8048c22c8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 02:01:37.082072 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling FlushMRSOp(4154be3f1f3c428580463c419d69892e): perf score=0.033364
I20240502 02:01:37.094743 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: FlushMRSOp(4154be3f1f3c428580463c419d69892e) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":236,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1342,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":462,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":197,"threads_started":2,"wal-append.queue_time_us":189}
I20240502 02:01:37.095634 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling UndoDeltaBlockGCOp(4154be3f1f3c428580463c419d69892e): 4581 bytes on disk
I20240502 02:01:37.096019 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: UndoDeltaBlockGCOp(4154be3f1f3c428580463c419d69892e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 02:01:37.134697 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:37.137858 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:37.139709 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:37.142555 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:37.144161 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:37.157231 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:37.172755 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:37.241647 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling FlushMRSOp(034cf1c3d47a4f7785c16d16815c229d): perf score=0.033396
I20240502 02:01:37.254222 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: FlushMRSOp(034cf1c3d47a4f7785c16d16815c229d) complete. Timing: real 0.012s user 0.008s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":243,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":2607,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":444,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":201,"threads_started":2,"wal-append.queue_time_us":170}
I20240502 02:01:37.255194 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling FlushMRSOp(dcc5daef41bc44c4855f2bcb44e1c6d5): perf score=0.033344
W20240502 02:01:37.259943 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 02:01:37.267766 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: FlushMRSOp(dcc5daef41bc44c4855f2bcb44e1c6d5) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1280,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":408,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":74,"threads_started":1,"wal-append.queue_time_us":209}
I20240502 02:01:37.268530 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling UndoDeltaBlockGCOp(034cf1c3d47a4f7785c16d16815c229d): 4921 bytes on disk
I20240502 02:01:37.268975 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: UndoDeltaBlockGCOp(034cf1c3d47a4f7785c16d16815c229d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240502 02:01:37.269620 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling UndoDeltaBlockGCOp(dcc5daef41bc44c4855f2bcb44e1c6d5): 4921 bytes on disk
I20240502 02:01:37.270004 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: UndoDeltaBlockGCOp(dcc5daef41bc44c4855f2bcb44e1c6d5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240502 02:01:37.365535 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:01:37.468782 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:37.475965 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:37.477510 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:37.479033 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:37.481487 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:37.481904 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:37.482463 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:37.482486 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:37.800689 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:37.806128 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:37.808877 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:37.810524 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:37.812402 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:37.818970 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:37.833212 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:37.912884 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:37.913523 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:37.949402 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:37.990789 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:01:38.135587 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:38.140751 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:38.142937 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:38.146569 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:38.150586 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:38.159559 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:38.167670 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:38.214901 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:01:38.299823 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:01:38.318643 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling FlushMRSOp(44c55b199a24432e902cce9121fc0780): perf score=0.033360
I20240502 02:01:38.330718 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: FlushMRSOp(44c55b199a24432e902cce9121fc0780) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":191,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1448,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":490,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":153,"threads_started":2,"wal-append.queue_time_us":191}
I20240502 02:01:38.331480 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling UndoDeltaBlockGCOp(44c55b199a24432e902cce9121fc0780): 4920 bytes on disk
I20240502 02:01:38.331892 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: UndoDeltaBlockGCOp(44c55b199a24432e902cce9121fc0780) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240502 02:01:38.460497 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:38.465512 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:38.467239 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:38.473543 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:38.480476 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:38.483498 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:38.486619 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:38.493530 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:38.530633 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:01:38.753989 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:01:38.799819 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:38.800103 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:01:38.801569 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:38.803185 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:38.804791 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:38.820927 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:38.821202 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:38.821224 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:38.877117 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:01:38.975545 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling FlushMRSOp(47a276e9780e448a9b3c6eab67925514): perf score=0.033341
I20240502 02:01:38.988102 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: FlushMRSOp(47a276e9780e448a9b3c6eab67925514) complete. Timing: real 0.012s user 0.008s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":183,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1585,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":448,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":1280,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":141}
I20240502 02:01:38.988873 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling UndoDeltaBlockGCOp(47a276e9780e448a9b3c6eab67925514): 4581 bytes on disk
I20240502 02:01:38.989244 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: UndoDeltaBlockGCOp(47a276e9780e448a9b3c6eab67925514) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240502 02:01:38.998241 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:39.049314 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:01:39.134191 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:39.136278 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:39.142104 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:39.142560 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:39.142567 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:39.159962 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:39.160444 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:39.184266 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:01:39.186225 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:01:39.240429 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling FlushMRSOp(d4d429fe60e74108bc0c9ca8509365f2): perf score=0.033389
I20240502 02:01:39.251554 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: FlushMRSOp(d4d429fe60e74108bc0c9ca8509365f2) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":65,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1434,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":440,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":96,"threads_started":1,"wal-append.queue_time_us":176}
I20240502 02:01:39.252419 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling FlushMRSOp(e6ac26fa6de44f7ca68e6c0b6b959db7): perf score=0.033362
I20240502 02:01:39.264642 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: FlushMRSOp(e6ac26fa6de44f7ca68e6c0b6b959db7) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":86,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":1378,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":421,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":121,"threads_started":1,"wal-append.queue_time_us":207}
I20240502 02:01:39.265457 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling UndoDeltaBlockGCOp(e6ac26fa6de44f7ca68e6c0b6b959db7): 4835 bytes on disk
I20240502 02:01:39.265864 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: UndoDeltaBlockGCOp(e6ac26fa6de44f7ca68e6c0b6b959db7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240502 02:01:39.266690 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling UndoDeltaBlockGCOp(d4d429fe60e74108bc0c9ca8509365f2): 4581 bytes on disk
I20240502 02:01:39.267091 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: UndoDeltaBlockGCOp(d4d429fe60e74108bc0c9ca8509365f2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240502 02:01:39.267643 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling FlushMRSOp(9313079c48c94ab1b7e022a91befafcd): perf score=0.033334
I20240502 02:01:39.279963 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: FlushMRSOp(9313079c48c94ab1b7e022a91befafcd) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1304,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":433,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":146,"threads_started":1,"wal-append.queue_time_us":262}
I20240502 02:01:39.280684 7229 maintenance_manager.cc:419] P 86bc8001c3464b929cd5c873861baa0c: Scheduling UndoDeltaBlockGCOp(9313079c48c94ab1b7e022a91befafcd): 4834 bytes on disk
I20240502 02:01:39.281038 7118 maintenance_manager.cc:643] P 86bc8001c3464b929cd5c873861baa0c: UndoDeltaBlockGCOp(9313079c48c94ab1b7e022a91befafcd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240502 02:01:39.313170 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling FlushMRSOp(e6ac26fa6de44f7ca68e6c0b6b959db7): perf score=0.033383
I20240502 02:01:39.325073 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: FlushMRSOp(e6ac26fa6de44f7ca68e6c0b6b959db7) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":1408,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":420,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":156,"threads_started":2,"wal-append.queue_time_us":156}
I20240502 02:01:39.325871 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling UndoDeltaBlockGCOp(e6ac26fa6de44f7ca68e6c0b6b959db7): 4835 bytes on disk
I20240502 02:01:39.326231 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: UndoDeltaBlockGCOp(e6ac26fa6de44f7ca68e6c0b6b959db7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240502 02:01:39.326858 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling FlushMRSOp(9f6c07e3923d47f29ef32561d7f438e1): perf score=0.033362
I20240502 02:01:39.338907 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: FlushMRSOp(9f6c07e3923d47f29ef32561d7f438e1) complete. Timing: real 0.012s user 0.010s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1415,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":479,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":89,"threads_started":1,"wal-append.queue_time_us":163}
I20240502 02:01:39.339646 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling UndoDeltaBlockGCOp(9f6c07e3923d47f29ef32561d7f438e1): 4921 bytes on disk
I20240502 02:01:39.340008 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: UndoDeltaBlockGCOp(9f6c07e3923d47f29ef32561d7f438e1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240502 02:01:39.468727 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:39.471266 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:39.476408 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:39.477209 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:39.477912 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:39.489180 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:39.498960 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:39.585484 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 02:01:39.606902 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling FlushMRSOp(a66b932d884e4df8a78841103d88ded0): perf score=0.033377
W20240502 02:01:39.618937 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:39.621735 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: FlushMRSOp(a66b932d884e4df8a78841103d88ded0) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":223,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1617,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":514,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":182,"threads_started":2,"wal-append.queue_time_us":173}
I20240502 02:01:39.622581 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling UndoDeltaBlockGCOp(a66b932d884e4df8a78841103d88ded0): 4834 bytes on disk
I20240502 02:01:39.622951 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: UndoDeltaBlockGCOp(a66b932d884e4df8a78841103d88ded0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 02:01:39.803957 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:39.804030 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:39.804220 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:39.804282 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:39.815860 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:39.821277 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:39.829044 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:40.104266 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling FlushMRSOp(7fa358140eb742b587a4d6a203738b4f): perf score=0.033351
I20240502 02:01:40.123726 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: FlushMRSOp(7fa358140eb742b587a4d6a203738b4f) complete. Timing: real 0.019s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3407,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":2771,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":642,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":210,"threads_started":2,"wal-append.queue_time_us":169}
I20240502 02:01:40.124852 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling UndoDeltaBlockGCOp(7fa358140eb742b587a4d6a203738b4f): 4581 bytes on disk
I20240502 02:01:40.126020 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: UndoDeltaBlockGCOp(7fa358140eb742b587a4d6a203738b4f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240502 02:01:40.136899 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:40.139150 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:40.142902 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:40.147830 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:40.148223 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:40.151024 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:40.166312 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 02:01:40.172030 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:40.235491 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:40.464243 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:40.465265 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:40.467198 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:40.474169 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:40.474937 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:40.475016 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:40.498870 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:40.674876 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:01:40.737849 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:01:40.785226 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:40.793773 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:40.795353 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:40.802556 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:40.808789 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:40.812112 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:40.816027 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:40.826478 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:40.835546 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:01:40.867173 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:01:41.062469 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 02:01:41.120195 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:41.121135 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:41.122424 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:41.123354 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:41.142093 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:41.154867 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:41.158916 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:41.189016 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:01:41.255180 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:41.364799 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:01:41.429948 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 02:01:41.452983 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:41.453809 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:41.457237 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:41.463074 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:41.471289 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:41.475092 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:41.495296 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:41.532632 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling FlushMRSOp(a0548274c47148d9b611fd3c978a0c34): perf score=0.033351
I20240502 02:01:41.544615 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: FlushMRSOp(a0548274c47148d9b611fd3c978a0c34) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":1594,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":495,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":137,"threads_started":2,"wal-append.queue_time_us":138}
I20240502 02:01:41.545323 8951 maintenance_manager.cc:419] P 070558c278e0457b9cc070ac0d662b5d: Scheduling UndoDeltaBlockGCOp(a0548274c47148d9b611fd3c978a0c34): 4581 bytes on disk
I20240502 02:01:41.545693 8841 maintenance_manager.cc:643] P 070558c278e0457b9cc070ac0d662b5d: UndoDeltaBlockGCOp(a0548274c47148d9b611fd3c978a0c34) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240502 02:01:41.609719 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:01:41.612650 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:01:41.631433 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:41.729982 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 02:01:41.788491 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:41.790213 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:41.791196 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:41.805039 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:41.805310 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:41.805850 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:41.806881 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:41.815888 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:01:42.110886 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:42.112552 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:42.117224 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:42.119740 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:42.121374 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:42.127431 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:42.137701 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:42.138597 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:42.286298 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 02:01:42.456763 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:42.457278 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:42.457454 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:42.458415 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:42.470825 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:42.472199 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:42.486635 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:42.531409 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:42.594704 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling FlushMRSOp(034cf1c3d47a4f7785c16d16815c229d): perf score=0.033375
I20240502 02:01:42.606871 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: FlushMRSOp(034cf1c3d47a4f7785c16d16815c229d) complete. Timing: real 0.012s user 0.011s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":231,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1613,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":453,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":153,"threads_started":2,"wal-append.queue_time_us":141}
I20240502 02:01:42.607652 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling UndoDeltaBlockGCOp(034cf1c3d47a4f7785c16d16815c229d): 4921 bytes on disk
I20240502 02:01:42.608013 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: UndoDeltaBlockGCOp(034cf1c3d47a4f7785c16d16815c229d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240502 02:01:42.763537 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:01:42.780081 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:42.780359 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:42.782052 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:42.787448 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:42.790524 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:42.802193 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:42.814007 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:42.920924 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:42.921487 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:42.972618 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 02:01:43.107118 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:43.108211 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:43.108505 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:43.109320 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:43.109695 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:43.111191 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:43.118589 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:43.133644 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling FlushMRSOp(a0548274c47148d9b611fd3c978a0c34): perf score=0.033369
I20240502 02:01:43.159303 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: FlushMRSOp(a0548274c47148d9b611fd3c978a0c34) complete. Timing: real 0.025s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":11628,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":1600,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":533,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":8427}
I20240502 02:01:43.160192 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling UndoDeltaBlockGCOp(a0548274c47148d9b611fd3c978a0c34): 4581 bytes on disk
I20240502 02:01:43.160657 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: UndoDeltaBlockGCOp(a0548274c47148d9b611fd3c978a0c34) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240502 02:01:43.193660 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:01:43.211658 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:01:43.227187 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:01:43.420920 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:01:43.445070 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:43.447340 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:43.451978 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:43.454466 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:43.463645 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:43.473666 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:43.482359 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:43.509371 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:43.552317 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:01:43.567718 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:01:43.767347 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:43.779227 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:43.786921 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:43.792933 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:43.800880 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:43.803596 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:43.810575 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:44.023761 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:01:44.040191 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:44.084929 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:01:44.112844 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:44.115366 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:44.124208 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:44.129516 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:44.134562 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:44.136878 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:44.158316 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:44.244132 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:01:44.263141 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:01:44.444324 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:44.449901 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:44.454906 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:44.459390 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:44.465822 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:44.467836 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 02:01:44.469372 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:44.470788 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:44.472959 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:44.776044 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:44.778468 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:44.784329 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:44.790784 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:44.796397 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:44.805652 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:44.808753 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:44.920205 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:44.961247 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 02:01:45.110652 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:45.112120 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:45.113411 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:45.116127 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:45.122815 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:45.123181 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:45.129949 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:45.437829 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:45.439580 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:45.443351 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:45.450466 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:45.459868 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:45.467504 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:45.469480 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:45.518772 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:45.646621 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:01:45.655339 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:01:45.769732 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:45.775178 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:45.778064 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:45.780968 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:45.784389 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:45.794559 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:45.806375 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:45.946484 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:01:46.027807 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:01:46.031270 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:01:46.077275 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:46.099145 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 02:01:46.104671 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:46.108366 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:46.111011 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:46.116637 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:46.118913 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:46.124238 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 02:01:46.135402 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:46.135787 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:46.435883 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:46.437858 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:46.438022 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:46.441344 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:46.448486 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:46.449473 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:46.462018 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:46.610277 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:01:46.629417 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:01:46.651983 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:46.768419 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:46.774600 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:46.774878 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:46.786612 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:46.787509 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:46.802112 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:46.811496 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:46.840520 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:01:46.861351 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:01:46.915949 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 02:01:47.103400 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:47.110639 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:47.115029 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:47.122351 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:47.124011 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:47.126932 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:47.130578 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:47.133152 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:47.444041 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:47.446756 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:47.453155 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:47.453984 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:47.462867 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:47.470067 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:47.470325 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:47.550194 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:47.574950 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:01:47.575601 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 02:01:47.776232 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:47.779219 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:47.782024 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:47.785111 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:47.788188 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:47.811400 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:47.814478 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:47.942977 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:47.943630 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:47.955315 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:01:47.980384 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:01:48.006238 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 02:01:48.112558 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:48.118953 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:48.119333 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:48.121345 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:48.122195 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:48.137316 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:48.140287 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:48.371408 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 02:01:48.435961 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:48.438876 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:48.441115 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:48.445191 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:48.462080 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:48.471050 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:48.471879 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:48.540668 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:48.607508 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:01:48.691206 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 02:01:48.767627 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:48.768150 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:48.768919 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:48.770287 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:48.781826 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:48.796563 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:48.806644 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:48.856909 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:01:49.075338 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:49.107539 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:49.107697 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:49.108651 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:49.117724 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:49.123188 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:49.124889 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:49.132764 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:49.227517 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:01:49.345249 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:01:49.369331 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:01:49.401427 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 02:01:49.440696 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:49.443315 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:49.443579 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:49.450569 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:49.458577 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:49.465982 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:49.471813 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:49.504313 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:01:49.699599 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:49.768628 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:49.772964 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:49.773684 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:49.783713 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:49.786284 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:49.792146 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:49.794236 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:50.014647 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 02:01:50.099632 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:50.101069 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:50.106065 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:50.108700 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:50.109277 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:50.110044 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:50.123549 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:50.291776 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:50.355846 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 02:01:50.429858 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:50.430119 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:50.441318 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:50.444089 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:50.446580 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:50.452574 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:50.463763 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:50.520566 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:01:50.555169 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:01:50.734968 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:50.758739 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:50.761461 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:50.763562 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:50.769048 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:50.773211 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:50.775871 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:50.790266 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:50.992223 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:01:51.074155 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 02:01:51.086575 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:51.089048 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:51.091051 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:51.091775 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:51.094795 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:51.102737 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:51.105996 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:51.137161 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:01:51.278255 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:01:51.297603 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:51.411715 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:51.419080 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:51.419296 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:51.423413 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:51.424005 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:51.431511 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:51.438575 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:51.716945 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 02:01:51.744890 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:51.745858 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:51.748759 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:51.759330 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:51.761961 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:51.765743 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:51.769116 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:51.790081 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:51.790767 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:51.904664 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:01:51.940347 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:01:51.980500 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 02:01:52.069736 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:52.074491 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:52.081671 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:52.083534 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:52.095168 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:52.105103 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:52.114966 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:52.309051 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:52.365247 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:52.416971 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:52.420907 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:52.423341 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:52.425451 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:52.427738 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:52.430766 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:52.440179 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:52.588690 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 02:01:52.748183 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:52.748270 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:52.749351 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:52.749630 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:52.760561 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:52.766424 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:52.767578 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:52.823514 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:52.824894 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:01:52.906836 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:53.003947 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 02:01:53.064852 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 02:01:53.085580 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:53.087046 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:53.091747 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:53.098335 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:53.103372 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:53.112138 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:53.112488 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:53.357115 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:01:53.423257 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:53.425930 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:53.427369 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:53.428625 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:53.430380 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:53.431461 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:53.443399 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:53.464632 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:53.634099 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:53.639685 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:53.648149 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 02:01:53.754684 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:53.754791 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:53.759801 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:53.771714 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:53.772104 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:53.774739 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:53.780588 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:53.866621 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:54.087491 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:54.094138 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:54.096079 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:54.096576 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:54.096772 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:54.111003 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:54.120877 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:54.364836 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:54.374846 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:54.393357 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:54.433243 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:54.433423 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:54.434309 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:54.435671 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:54.438807 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:54.442623 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:54.446844 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:54.459355 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:54.645576 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 02:01:54.760529 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:54.760605 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:54.765151 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:54.767284 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:54.767745 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:54.775712 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:54.790244 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:54.790589 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:54.806982 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:01:54.852088 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:55.092799 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:55.093176 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:55.097036 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:55.097306 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:55.111388 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:55.116307 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:55.126523 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:55.308641 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 02:01:55.420022 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:55.422277 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:55.424204 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:55.429615 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:55.430694 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:55.432502 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:55.434897 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:55.452718 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:55.476857 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:55.670892 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 02:01:55.750512 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:55.750825 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:55.752259 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:55.754170 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:55.754933 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:55.755436 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:55.755566 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:56.026871 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:56.028946 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:56.081215 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:56.082859 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:56.083530 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:56.083727 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:56.093173 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:56.095712 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:56.096892 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:56.143908 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:56.267004 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:01:56.267644 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 02:01:56.409950 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:56.410488 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:56.411922 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:56.415521 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:56.417793 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:56.421963 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:56.422443 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:56.618520 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:56.743047 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:56.747195 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:56.753441 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:56.756660 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:56.762488 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:56.762672 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:56.769119 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:56.835281 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:56.930147 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:56.983675 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:57.013267 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:57.044984 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 02:01:57.089617 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:57.091887 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:57.101436 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:57.102806 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:57.103269 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:57.104880 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:57.120044 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:57.126333 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:57.356472 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 02:01:57.426101 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:57.426223 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:57.427742 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:57.443441 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:57.443549 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:57.443645 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:57.448318 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:57.577690 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:57.596729 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 02:01:57.770006 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:57.775486 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:57.777716 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:57.778065 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:57.778337 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:57.787663 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:57.804504 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:57.836889 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:58.032444 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:01:58.034901 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 02:01:58.099455 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:58.105600 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:58.105669 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:58.112854 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:58.129904 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:58.131373 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:58.131677 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:01:58.170047 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 02:01:58.170786 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:58.403085 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 02:01:58.438781 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:58.440192 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:58.442528 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:58.444797 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:58.445056 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:58.457403 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:58.472460 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:58.613906 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:01:58.667125 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:01:58.765475 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:58.775741 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:58.775979 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:58.779100 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:58.782889 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:58.795166 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:58.802930 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:58.834241 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:58.895324 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 02:01:59.111078 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:59.111591 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:59.113636 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:59.122681 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:59.128697 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:59.129459 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:59.134769 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:01:59.147639 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:59.149401 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:59.377538 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 02:01:59.439574 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:59.441324 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:59.441596 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:59.443766 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:59.459005 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:59.465451 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:59.468936 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:01:59.518175 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:01:59.614557 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:01:59.616047 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:01:59.759747 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 02:01:59.765630 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:59.777709 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:59.778474 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:01:59.782848 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:59.785804 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:01:59.786222 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:59.811196 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:59.918037 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 02:02:00.096027 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:00.097069 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:00.102497 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:00.106631 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:00.118170 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:00.124857 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:00.126794 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:00.165283 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:00.345090 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 02:02:00.421866 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:00.425145 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:00.431674 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:00.435201 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:00.438494 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:00.439309 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:00.442541 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:00.521692 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:02:00.550266 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:02:00.704007 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:00.748944 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:00.749289 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:00.751132 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:00.754585 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:00.756076 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:00.768111 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:00.778481 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:01.080655 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:01.082712 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:01.084899 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:01.095090 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:01.095479 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:01.101567 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:01.109675 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 02:02:01.122228 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:01.182914 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:01.328161 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 02:02:01.409685 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:01.412812 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:01.415315 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:01.424578 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:01.429417 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:01.429895 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:01.432827 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:01.469878 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:02:01.565855 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:02:01.689402 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:01.738222 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:01.741333 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:01.741994 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:01.745098 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:01.746464 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 02:02:01.746546 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:01.747850 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:01.753639 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 02:02:01.771883 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:01.839550 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:02:02.055859 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:02:02.059773 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 02:02:02.067499 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:02.073501 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:02.078377 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:02.079159 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:02.090528 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:02.092113 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:02.092200 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:02.170168 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:02.401579 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:02.403163 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:02.404453 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:02.410550 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:02.414041 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:02.423009 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 02:02:02.433863 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:02.435058 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:02.635680 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:02.674420 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:02:02.691093 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 02:02:02.726121 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:02.726763 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:02.727509 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:02.728652 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:02.750638 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:02.751813 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:02.753252 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:02.900988 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:02:03.019165 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 02:02:03.049587 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:03.054994 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:03.055663 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:03.056361 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:03.057379 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:03.057603 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:03.081447 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:03.188598 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:02:03.207967 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:03.208616 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:03.376138 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:03.378628 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:03.382117 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:03.384696 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:03.390697 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:03.409291 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:03.410552 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:03.711555 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:03.713719 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:03.717783 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:03.723165 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:03.733354 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:03.742556 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:03.742870 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:03.788736 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:03.828253 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:02:03.888855 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 02:02:04.046886 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:04.047995 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:04.048887 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:04.048897 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:04.051190 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:04.070216 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:04.082502 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:04.109031 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:02:04.132190 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:02:04.198954 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:02:04.226382 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:02:04.285450 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:04.315871 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 02:02:04.373375 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:04.378775 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:04.389410 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:04.391265 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:04.393010 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:04.393201 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:04.408139 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:04.546030 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 02:02:04.702011 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:04.703197 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:04.703363 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:04.704596 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:04.705044 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:04.705479 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:04.705497 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:04.734473 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:04.980180 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 02:02:05.027714 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:05.033469 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:05.038218 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:05.041899 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:05.043258 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:05.046937 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:05.047005 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:05.142724 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:05.232007 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 02:02:05.354025 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:05.365208 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:05.367007 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:05.377231 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:05.377692 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:05.379423 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:05.387064 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:05.434454 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:02:05.579972 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:05.679268 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:05.685056 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:05.687781 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:05.696101 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:05.696367 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:05.697508 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:05.718468 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:05.812117 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:02:05.837345 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 02:02:06.003952 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:06.008791 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:06.018276 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:06.019735 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:06.020879 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:06.023224 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:06.031823 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:06.039887 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:06.335160 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:06.343050 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:06.349272 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:06.350561 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:06.351331 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:06.354349 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:06.360450 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:06.365002 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:02:06.432004 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:02:06.437989 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:02:06.447877 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:02:06.516697 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:06.671731 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:06.672224 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:06.672794 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:06.676713 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:06.682861 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:06.686256 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:06.703886 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:06.772591 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:02:06.812533 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:02:06.823958 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:02:06.873363 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:02:06.900859 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:06.995180 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:06.997128 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:07.005051 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:07.010921 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:07.011476 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:07.018501 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:07.019418 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:07.327461 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:07.329265 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:07.329357 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:07.331733 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:07.341153 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:07.341637 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:07.352046 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:07.416529 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:02:07.480330 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:07.662420 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:07.663847 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:07.667584 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:07.678893 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:07.681711 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:07.683290 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:07.690541 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:07.766072 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:02:07.795498 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:02:07.992194 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:07.993201 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:07.993927 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:07.994017 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:07.999114 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:08.004943 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:08.010249 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:08.010949 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:08.242062 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 02:02:08.314973 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:08.315251 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:08.317085 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:08.317303 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:08.318217 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 02:02:08.319072 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:08.319841 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:08.336067 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:08.386992 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:02:08.428754 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:02:08.517448 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:08.633832 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:08.638710 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:08.639288 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:08.647002 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:08.648618 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:08.657361 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:08.668221 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:08.770768 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:02:08.825534 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:02:08.935343 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:08.959659 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:08.964996 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:08.965204 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:08.965247 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:08.972692 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:08.989503 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:08.990911 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:09.114967 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:02:09.115710 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:02:09.141135 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:02:09.151338 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 02:02:09.287808 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:09.289109 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:09.293834 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:09.302661 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:09.304230 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:09.313956 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:09.322930 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:09.340030 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:09.386279 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:02:09.528725 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 02:02:09.619264 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:09.620118 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:09.623955 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:09.624426 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:09.628931 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:09.636023 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:09.647172 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:09.817175 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:09.895622 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 02:02:09.944135 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:09.945101 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:09.949296 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:09.961798 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:09.962114 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:09.965075 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:09.988507 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:10.123356 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 02:02:10.280408 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:10.281637 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:10.283283 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:10.284324 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:10.285759 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:10.294540 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:10.309291 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:10.365904 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:10.483599 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 02:02:10.602835 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:10.604508 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:10.609349 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:10.610922 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:10.614758 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:10.619952 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:10.634999 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:10.870177 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:02:10.872547 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:02:10.905403 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:10.933816 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:10.934655 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:10.937769 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:10.941040 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:10.950888 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:10.951155 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:10.951638 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:11.223666 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 02:02:11.266942 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:11.271451 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:11.271482 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:11.273689 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:11.274112 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:11.278594 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:11.282550 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:11.364337 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:02:11.368167 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:02:11.420969 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:02:11.470966 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:02:11.522212 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:11.591179 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:11.593920 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:11.597189 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:11.609958 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:11.611852 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:11.614042 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:11.622856 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:11.710916 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:02:11.771339 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 02:02:11.918805 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:11.920866 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:11.925029 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:11.931685 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:11.934062 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:11.953999 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:11.954511 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:11.984575 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:12.247679 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:12.248502 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:12.261410 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:12.265129 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:12.265604 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:12.272640 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:12.275360 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:12.360591 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:02:12.371093 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:02:12.374660 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:12.388864 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 02:02:12.574126 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:12.575346 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:12.579293 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:12.582752 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:12.586566 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:12.588258 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:12.603940 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:12.865883 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:12.878264 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 02:02:12.897015 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:12.898608 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:12.899989 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:12.900226 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:12.911252 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:12.924572 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:12.924968 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:12.961663 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 02:02:13.223140 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:13.227491 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:13.228058 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:13.228657 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:13.229369 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:13.232666 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:13.250738 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:13.355152 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:02:13.403378 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:13.404023 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:13.429495 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 02:02:13.546679 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:13.549418 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:13.552011 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:13.556617 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:13.568921 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:13.569305 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:13.571899 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:13.684494 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:02:13.858671 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:13.872937 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:13.873485 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:13.874007 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:13.875535 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:13.884755 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:13.885154 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:13.885766 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:13.903512 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:02:13.972945 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:02:14.032930 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:02:14.132220 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 02:02:14.201638 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:14.202775 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:14.205808 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:14.207368 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:14.209964 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:14.222013 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:14.230957 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:14.248651 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:14.466094 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 02:02:14.527144 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:14.528903 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:14.535943 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:14.537339 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:14.539362 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:14.545121 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:14.547354 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:14.615679 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:02:14.688647 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:02:14.851333 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:02:14.857321 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:14.871614 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:14.871884 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:14.872140 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:14.874106 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:14.874687 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:14.877894 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:14.891646 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:15.044275 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 02:02:15.191766 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:15.192462 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:15.205776 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:15.207301 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:15.209372 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:15.209385 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:15.210014 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:15.367722 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:15.517156 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:15.517997 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:15.518599 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:15.522578 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:15.533556 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:15.534368 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:15.537123 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:15.686096 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 02:02:15.841779 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:15.842600 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:15.846027 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:15.852887 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:15.855170 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:15.867798 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:15.880407 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:15.937265 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:02:15.963420 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:16.066041 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:02:16.161796 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 02:02:16.174186 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:16.179373 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:16.179791 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:16.182112 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:16.187043 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:16.189723 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:16.191419 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:16.506142 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:16.508713 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:16.516091 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:16.516441 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:16.517398 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:16.528467 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:16.535414 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:16.536566 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:02:16.557690 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:02:16.583235 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:16.670994 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:02:16.734822 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 02:02:16.830806 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:16.831552 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:16.838270 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 02:02:16.849431 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:16.851735 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:16.852217 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:16.853807 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:16.857616 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:16.998019 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:17.003042 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:02:17.096712 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 02:02:17.180756 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:17.181560 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:17.181838 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:17.182119 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:17.185525 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:17.186615 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:17.197424 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:17.390782 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:02:17.480835 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 02:02:17.510370 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:17.510769 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:17.511369 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:17.518921 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:17.526995 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:17.540071 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:17.544281 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:17.608558 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:17.841281 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:17.844704 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:17.846581 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:17.849341 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:17.849895 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:17.854326 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:17.863281 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:18.035480 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:02:18.053783 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:18.178223 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:18.178545 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:18.179056 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:18.182775 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:18.195341 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 02:02:18.198786 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:18.209671 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:18.210580 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:18.290122 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:02:18.371773 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 02:02:18.524852 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:18.525697 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:18.532536 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:18.537781 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:18.541087 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:18.542291 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:18.554234 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:18.581853 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:18.653925 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:18.747759 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 02:02:18.854957 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:18.855334 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:18.855441 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:18.861088 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:18.861732 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:18.862597 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:18.873926 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:19.178550 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 02:02:19.180437 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:19.183966 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:19.189065 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:19.192623 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:19.202564 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:19.204016 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:19.211441 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:19.220908 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:19.287503 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:02:19.308846 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:02:19.321869 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:02:19.405298 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 02:02:19.518697 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:19.520071 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:19.525030 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:19.527143 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:19.528443 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:19.534584 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:19.535087 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:19.635929 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:02:19.669276 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:19.701400 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 02:02:19.844781 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:19.847596 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:19.849128 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:19.851135 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:19.861402 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:19.862257 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:19.862603 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:20.067445 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:02:20.077178 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 02:02:20.173552 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:20.176066 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:20.178335 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:20.183931 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:20.186563 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:20.188012 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:20.188161 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:20.267074 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:20.503671 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:20.509773 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:20.510732 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:20.513111 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:20.521791 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:20.523512 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:20.523877 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:20.576427 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:02:20.664412 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:02:20.763708 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:20.764004 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 02:02:20.833014 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:20.833705 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:20.835124 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:20.837266 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:20.840452 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:20.843114 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:20.849478 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:21.051831 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 02:02:21.164539 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:21.165222 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:21.165971 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:21.166267 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:21.167280 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:21.173933 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:21.196589 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:21.335072 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:02:21.367074 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:21.433573 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:02:21.461531 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 02:02:21.493017 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:21.495496 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:21.499351 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:21.507711 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:21.509814 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:21.511071 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:21.518055 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:21.592325 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:02:21.811090 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:21.817584 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:21.819828 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:21.824229 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:21.832038 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:21.839776 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:21.840840 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:21.853214 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:22.030053 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:02:22.056020 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:02:22.107707 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 02:02:22.149957 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:22.154924 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:22.156430 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:22.157016 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:22.168236 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:22.170038 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:22.170161 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:22.315889 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:02:22.335963 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:22.486572 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:22.490537 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:22.492931 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:22.494081 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:22.498239 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:22.498600 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:22.517957 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:22.767201 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 02:02:22.813427 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:22.815217 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:22.818118 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:22.823180 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:22.833276 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:22.837625 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:22.843258 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:22.848224 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:22.900583 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:02:23.089284 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 02:02:23.148162 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:23.151387 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:23.151733 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:23.152402 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:23.158020 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:23.175050 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:23.175699 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:23.178174 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:02:23.301676 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:02:23.330003 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:23.396543 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 02:02:23.474635 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:23.475203 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:23.475641 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:23.477363 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:23.484611 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:23.485710 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:23.518947 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:23.782951 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:02:23.812750 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:23.814182 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:23.814847 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:23.834632 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:23.837723 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:23.839319 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:23.848661 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:23.912657 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:23.919431 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:02:23.941812 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:02:23.943028 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 02:02:24.146584 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:24.147974 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:24.149582 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:24.156373 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:24.159255 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:24.160395 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:24.163741 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:24.382833 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:02:24.399509 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:24.469942 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:24.473129 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:24.483701 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:24.484941 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:24.488269 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:24.489485 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:24.498790 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:24.534128 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:02:24.631328 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 02:02:24.814747 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:24.816459 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:24.816655 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:24.817286 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:24.818248 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:24.823489 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:24.824200 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:24.842322 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:02:24.863687 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:02:24.906298 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:25.049197 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 02:02:25.140460 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:25.145901 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:25.146044 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:25.150632 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:25.152449 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:25.152701 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:25.166230 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:25.292872 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:02:25.348639 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 02:02:25.470953 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:25.472781 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:25.477599 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:25.477784 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:25.478261 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:25.492014 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:25.498345 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:25.499293 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:25.805761 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:25.809234 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:25.810662 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:25.813700 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:25.820962 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:25.835253 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:25.844753 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:25.918582 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:02:25.966442 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:25.987640 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 02:02:26.139874 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:26.141371 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:26.147547 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:26.148068 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:26.150054 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:26.159013 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:26.175997 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:26.185961 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:02:26.442029 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:02:26.448889 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:26.469288 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:26.477710 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:26.480343 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:26.484228 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:26.485229 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:26.499583 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:26.513211 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:26.524962 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:02:26.723388 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:02:26.743685 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:02:26.744364 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 02:02:26.816182 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:26.820856 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:26.822933 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:26.824105 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:26.826599 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:26.843761 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:26.861047 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:27.035468 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:27.161679 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:27.163244 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:27.166891 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:27.167253 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:27.168880 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:27.172662 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:27.188558 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:27.326239 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:02:27.356247 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 02:02:27.495613 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:27.497156 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:27.501133 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:27.501538 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:27.502255 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:27.502575 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:27.511611 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:27.589411 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:27.595503 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 02:02:27.825999 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:27.828253 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:27.831334 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:27.837930 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:27.837947 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:27.838500 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:27.847453 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:27.887624 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:02:28.035648 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:02:28.138520 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:28.160470 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:28.161332 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:28.164127 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:28.165226 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:28.171548 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:28.179056 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:28.181672 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:28.339805 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:02:28.443688 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:02:28.499348 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 02:02:28.503914 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:28.509255 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:28.510115 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:28.514515 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:28.515271 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:28.517671 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:28.525911 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:28.665401 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:02:28.676007 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:28.724922 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 02:02:28.830917 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:28.832190 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:28.842603 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:28.845752 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:28.846172 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:28.856096 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:28.856333 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:29.087698 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:02:29.149804 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:02:29.154274 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:29.154961 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 02:02:29.160151 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:29.161623 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:29.161743 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:29.165175 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:29.167940 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:29.183467 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:29.186631 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:29.444689 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 02:02:29.486126 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:29.488266 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:29.488950 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:29.495189 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:29.504902 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:29.505440 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:29.511569 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:29.733824 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:02:29.773507 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:29.810125 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:29.812541 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:29.816550 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:29.817119 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:29.823429 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:29.823872 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:29.833911 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:29.851200 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:02:29.909623 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:02:29.919189 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 02:02:30.144372 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:30.145843 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:30.149945 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:30.157675 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:30.158581 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:30.159015 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:30.161502 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:30.223417 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:30.357107 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 02:02:30.466262 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:30.466728 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:30.472455 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:30.480295 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:30.486837 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 02:02:30.488595 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:30.496886 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:30.500820 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:30.805037 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:30.805533 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:30.806073 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:30.807698 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:30.809365 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:30.818351 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:30.827500 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:30.835776 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:30.906998 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:02:30.970963 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 02:02:31.132006 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:31.132061 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:31.137475 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:31.139215 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:31.142454 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:31.144845 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 02:02:31.150348 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:31.158950 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:31.340688 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:02:31.389420 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:31.453800 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:31.454353 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:31.455822 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:31.461167 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:31.462628 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:31.464280 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:31.464469 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 02:02:31.482982 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:31.729147 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:02:31.731180 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 02:02:31.780932 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:31.783885 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:31.788479 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:31.789953 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:31.798658 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:31.799127 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:31.800311 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:31.895812 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:31.901165 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 02:02:32.100420 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:32.105127 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:32.113629 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:32.115064 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:32.115090 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:32.118103 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:32.118541 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:32.165364 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:02:32.377344 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 02:02:32.434265 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:32.439410 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:32.440035 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:32.444145 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:32.447293 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:32.458891 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:32.472080 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:32.477921 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:32.535810 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:02:32.668012 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 02:02:32.763453 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:32.764694 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:32.765708 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:32.766000 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:32.767766 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:32.787345 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:32.795620 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:33.012933 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:02:33.038578 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:33.093127 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:33.093849 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:33.098358 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:33.099929 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:33.103377 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:33.111595 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:33.114492 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:33.324067 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 02:02:33.429544 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:33.429951 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:33.434581 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:33.443221 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:33.446054 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:33.447947 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:33.463194 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:33.513612 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:02:33.518093 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:02:33.573787 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:33.705435 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 02:02:33.753979 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:33.755003 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:33.755415 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:33.756369 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:33.761602 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:33.761824 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:33.778126 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:33.831898 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:02:33.993566 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:34.056901 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 02:02:34.090129 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:34.090459 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:34.091702 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:34.103904 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:34.104398 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:34.105036 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:34.114553 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:34.153151 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:02:34.205708 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:34.382918 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 02:02:34.421054 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:34.424593 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:34.428452 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:34.430927 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:34.431335 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:34.432788 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:34.450421 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:34.489452 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:34.747505 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:34.748418 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:34.750711 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:34.754541 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:34.763644 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:34.767755 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:34.768261 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:34.810570 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:02:34.839675 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:02:35.066360 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:35.067000 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 02:02:35.089298 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:35.090833 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:35.091459 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:35.093345 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:35.101419 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:35.110136 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:35.117426 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:35.300987 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:02:35.428000 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:35.433243 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:35.434065 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:35.440986 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:35.445263 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:35.453210 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:35.454231 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 02:02:35.459787 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:35.491403 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:35.693230 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 02:02:35.763003 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:35.764142 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:35.768973 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:35.773815 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:35.778882 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:35.779194 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:35.794620 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:35.876199 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:02:36.022869 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:02:36.075270 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:36.095438 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:36.101130 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:36.103853 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:36.105537 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:36.106561 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:36.120888 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:36.142838 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:36.269994 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:02:36.336277 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 02:02:36.437631 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:36.439476 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:36.443392 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:36.447937 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:36.450784 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:36.452970 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:36.462795 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:36.564661 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:02:36.582167 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:02:36.597894 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:36.770129 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:36.777130 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:36.779230 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:36.788606 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:36.798771 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:36.814721 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:36.815002 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:36.911553 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:02:36.912290 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:02:36.996591 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 02:02:37.113608 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:37.113706 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:37.119324 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:37.126581 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:37.136173 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 02:02:37.139266 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:37.139277 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:37.149590 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:37.157444 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:37.448673 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:37.448673 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:37.456007 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:37.459290 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:37.459364 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:37.471949 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:37.481984 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:37.754412 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:37.781961 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:37.783957 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:37.785697 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:37.786852 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:37.796268 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:37.797365 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:37.797685 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:37.913398 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:02:37.931591 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:02:38.095886 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 02:02:38.108700 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:38.109215 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:38.119683 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:38.120293 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:38.131654 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:38.136045 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:38.136673 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:38.166653 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:02:38.187927 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:38.396785 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 02:02:38.454180 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:38.456775 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:38.459369 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:38.466768 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:38.468587 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:38.469456 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:38.473690 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:38.577136 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:02:38.718335 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:38.776619 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 02:02:38.779199 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:38.780431 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:38.780764 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:38.794164 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:38.794749 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:02:38.798954 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:38.801584 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:38.805649 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:02:38.817281 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:38.961920 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 02:02:39.113356 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:39.114781 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:39.117218 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:39.118073 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:39.118347 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:39.124889 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:39.154230 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:39.218605 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:39.219297 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:39.259930 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:02:39.309146 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 02:02:39.443039 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:39.446923 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:39.447322 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:39.448959 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:39.451274 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:39.465062 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:39.466570 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:39.553081 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:02:39.687893 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:39.783509 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:39.786810 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:39.789350 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:39.798272 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:39.802314 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:39.807276 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:39.812317 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:40.113022 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:40.114606 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:40.117655 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:40.117950 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:40.119524 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:40.128072 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:40.131529 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:40.185415 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:40.200640 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:02:40.264539 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:02:40.438656 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:02:40.445582 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:40.446596 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:40.448010 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:40.449945 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:40.454455 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:40.464041 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:40.488739 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:40.489629 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:02:40.702320 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:40.784684 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:40.786046 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:40.788743 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:40.789942 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:40.789988 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 02:02:40.797001 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:40.801556 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:40.810247 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:40.979298 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:02:41.117322 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:41.118731 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:41.121675 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:41.123032 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:41.125625 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:41.134917 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:41.136198 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:41.205222 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:41.266568 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:02:41.413030 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 02:02:41.446336 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:41.447075 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:41.451503 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:41.458032 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:41.461297 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:02:41.461566 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:41.462987 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:41.468166 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:41.640908 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:02:41.657126 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:02:41.742018 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:41.774988 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:41.776288 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:41.778097 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:02:41.781384 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:41.787799 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:41.795773 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:41.797432 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:41.806815 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:41.929025 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:02:42.025465 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 02:02:42.110872 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:42.111721 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:42.114951 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:42.115594 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:42.115706 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:42.117544 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:42.127874 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:42.237689 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:42.442185 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:42.446261 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:42.453516 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:42.456254 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:42.458717 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:42.459123 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:42.489177 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:42.524403 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:02:42.670241 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:42.771292 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:42.777971 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:42.782546 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:42.789156 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:42.792224 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:42.792510 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:42.792516 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:42.876713 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:02:43.059403 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:02:43.102411 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:43.104228 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:43.108144 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:43.109544 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:43.110829 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:43.113368 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:43.142655 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:43.191910 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:43.332083 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 02:02:43.438488 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:43.438562 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:43.449030 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:43.449050 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:43.449326 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:43.461745 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:43.469522 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:02:43.481832 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:43.587282 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:02:43.724844 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:43.770964 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:43.775202 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:43.777364 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:43.777647 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:43.780948 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:43.782267 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:43.801797 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 02:02:43.809067 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:43.878454 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:02:43.960332 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:02:44.104604 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:44.105682 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:44.106344 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:44.107324 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 02:02:44.107977 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:44.110251 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:44.110641 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:44.110683 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:44.128604 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:44.175043 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:02:44.200389 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:02:44.239970 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:02:44.240535 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 02:02:44.430617 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:44.431871 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:44.439831 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:44.444977 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:44.446193 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:44.447712 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:44.459478 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:44.709146 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:44.751183 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:44.751287 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:44.753676 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:44.755146 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:44.763532 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:44.770022 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:44.774219 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:44.955391 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 02:02:45.074657 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:45.077231 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:45.091576 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:45.091795 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:45.095302 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:45.100884 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:45.104871 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:45.105378 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:45.252643 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:02:45.399729 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 02:02:45.407773 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:45.412211 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:45.413198 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:45.423696 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:45.426301 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:45.429391 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:45.447230 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:45.480850 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:45.515470 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 02:02:45.730221 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:45.731463 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:45.731873 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:45.738523 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:45.749645 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:45.762190 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:45.762248 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:45.776537 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:02:45.882962 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:46.062414 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:46.062664 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:46.063138 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:46.065202 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:46.068290 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:46.079442 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:46.094918 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:46.116428 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:02:46.247669 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:02:46.337859 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:46.384080 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 02:02:46.388070 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:46.392117 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:46.401574 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:46.402669 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:46.408690 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:46.410166 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:46.425066 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:46.432397 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:02:46.479012 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:02:46.660650 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:02:46.669924 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 02:02:46.714722 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:46.715023 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:46.718780 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:46.727036 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:46.729032 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:46.741472 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:46.751924 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:46.786306 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:02:46.788597 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:47.036770 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 02:02:47.046883 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:47.047051 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:47.055630 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:47.056797 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:47.057452 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:47.067457 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:47.091825 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:47.218842 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:02:47.359560 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:47.388368 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:47.390221 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:47.391290 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:47.395839 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:47.397579 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:47.408700 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:47.420965 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:47.709534 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:47.710263 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:47.710816 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:47.715387 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:47.736229 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:47.737493 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:47.738232 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:47.779424 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:47.862866 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:02:48.009112 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 02:02:48.025446 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:48.027468 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:48.038061 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:48.039103 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:48.041371 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:48.045879 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:48.050946 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:48.324503 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 02:02:48.350044 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:48.350631 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:48.354565 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:48.362608 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:48.362883 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:48.370010 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:48.377712 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:48.398622 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:48.687593 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:48.688650 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:48.689229 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:48.692931 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:48.693179 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:48.699441 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:48.707974 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:48.740722 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:02:48.780956 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:02:48.796744 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:02:48.814258 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:02:48.861869 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:48.913780 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 02:02:49.012246 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:49.014680 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:49.016443 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:49.017179 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:49.018527 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:49.018640 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:49.021776 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:49.109757 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:02:49.131453 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:02:49.153381 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 02:02:49.333133 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:49.338501 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:49.340515 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:49.343325 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:49.343741 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:49.348600 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:49.370080 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:49.371210 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:02:49.372360 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:49.405975 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 02:02:49.674221 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:49.676554 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:49.677278 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:49.679432 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:49.681974 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:49.692154 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:49.706306 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:49.716028 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:02:49.853518 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:49.995848 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:49.997740 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:49.999336 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:50.003773 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:50.010561 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:50.014550 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:50.032146 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:50.245499 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:50.321303 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:50.326968 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:50.333048 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:50.333644 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:50.335829 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:50.336664 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 02:02:50.336715 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:50.348632 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:50.514223 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 02:02:50.647707 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:50.649441 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:50.651316 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:50.651829 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:50.664896 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:50.666980 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:50.668170 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:50.741797 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:02:50.842978 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:50.873797 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:02:50.874541 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 02:02:50.971125 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:50.973276 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:50.977914 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:50.979331 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:50.982889 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:50.983582 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:50.986333 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:51.217082 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 02:02:51.299847 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:51.301090 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:51.302376 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:51.304651 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:51.311326 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:51.323676 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:51.333309 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:51.380491 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:51.400450 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:02:51.410346 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 02:02:51.626972 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:51.627729 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:51.631268 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:51.643664 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:51.646145 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 02:02:51.646525 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:51.647393 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:51.654556 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:51.708926 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:02:51.848181 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:51.916883 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 02:02:51.961197 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:51.962494 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:51.962635 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:51.965198 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:51.972618 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:51.973121 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:51.981508 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:52.071900 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:02:52.121734 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:02:52.198319 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 02:02:52.282681 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:52.286548 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:52.287648 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:52.289578 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:52.291591 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:52.292181 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:52.292949 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:52.306715 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:52.612485 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:52.617343 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:52.617527 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:52.619401 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:52.619634 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:52.621062 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:52.649627 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:52.765331 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:52.857156 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 02:02:52.935652 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:52.937027 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:52.941989 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:52.948009 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:52.957880 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:52.964226 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:52.970890 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:53.093780 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:53.096101 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 02:02:53.263203 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:53.263927 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:53.266254 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:53.272178 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:53.285250 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:53.286478 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:53.298914 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:53.309625 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:53.335593 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:53.589993 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:53.590385 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:53.592258 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:53.594090 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:53.612825 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:53.615525 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:53.623620 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:53.661263 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:53.710841 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:53.833662 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:53.837606 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 02:02:53.922178 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:53.922943 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:53.923940 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:53.931756 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:53.939138 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:53.941910 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:53.958067 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:54.046463 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:02:54.230576 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 02:02:54.253082 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:54.254467 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:54.256323 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:54.265769 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:54.273102 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:54.277863 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:54.290143 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:54.333047 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:54.424485 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:54.529121 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:02:54.529904 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:54.567127 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 02:02:54.581945 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:54.586977 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:54.588546 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:54.591542 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:54.599512 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:54.600622 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:54.601770 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 02:02:54.613143 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:54.779167 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:54.911269 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:54.912715 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:54.915460 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:54.918102 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:54.925227 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:54.926254 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:54.931552 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:55.234733 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:55.239693 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:55.239763 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:55.247961 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:55.252784 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:55.261921 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:55.267572 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:55.313594 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:55.389155 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:55.457379 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 02:02:55.570696 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:55.571023 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:55.572763 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:55.574095 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:55.578774 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:55.587194 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:55.592152 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:55.599378 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:55.789577 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:55.897392 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:55.901094 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:55.902102 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:55.903244 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:55.907210 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:55.907485 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:55.928968 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:55.963337 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:02:55.963996 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:56.007922 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 02:02:56.231717 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:56.233480 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:56.236059 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:56.239382 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:56.239871 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:56.253266 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:56.254657 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:56.344465 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:56.368554 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:56.457146 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 02:02:56.581398 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:56.585217 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:56.587256 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:56.589040 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:56.590934 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:56.602041 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:56.606598 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:56.667264 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:56.682597 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:02:56.767711 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:56.898368 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 02:02:56.912003 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:56.913455 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 02:02:56.914971 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:56.915927 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:56.921515 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:56.923122 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:56.924988 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:56.925843 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:57.135496 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:02:57.215009 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:57.243237 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:57.248490 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:57.250795 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:57.252677 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:57.254900 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:57.258574 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:57.278487 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:57.285674 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 02:02:57.567960 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:57.579059 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:57.579504 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:57.579937 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:57.580760 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:57.581873 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:57.582056 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:02:57.644759 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:57.737694 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:02:57.897030 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 02:02:57.904909 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:57.904970 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:57.909947 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:57.917713 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:57.927210 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:57.927366 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:57.929100 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:58.104362 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:58.228451 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:58.228497 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:58.234869 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:58.235713 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:58.238143 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:58.250944 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:58.251528 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:58.303632 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:58.526562 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:02:58.543432 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:58.548997 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:58.550304 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:58.550791 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:58.554221 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:58.571147 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:58.589056 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:02:58.642201 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:58.659128 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 02:02:58.872193 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:58.872845 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:58.875135 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:58.878707 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:02:58.881240 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:58.882565 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:58.895233 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:58.921653 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:02:58.995712 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:59.085749 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:02:59.118299 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:59.186446 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 02:02:59.190893 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:59.198176 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:59.199312 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:59.200124 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:59.202188 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:59.214855 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:02:59.223531 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 02:02:59.232659 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:59.401721 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:02:59.493640 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:02:59.502219 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:02:59.519181 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:59.519348 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:59.522616 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:59.525794 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:59.530238 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:02:59.536798 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:59.545817 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:02:59.674232 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:02:59.698330 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 02:02:59.845336 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:59.847901 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:59.852578 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:02:59.857977 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:59.860163 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:59.872723 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:02:59.877899 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:02:59.887190 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:00.169903 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:03:00.173424 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:00.178290 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:00.181041 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:00.191006 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:00.196910 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:00.210263 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:00.210263 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:00.379199 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:00.414172 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 02:03:00.502564 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:00.503973 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:00.506474 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:00.507553 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:00.508544 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:00.511404 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:00.513494 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:00.590773 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:03:00.825433 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:00.826596 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:00.829034 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:00.829854 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:00.843430 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:00.843915 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:00.845947 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:01.002653 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:01.143726 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:03:01.148505 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 02:03:01.148864 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:01.151616 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:01.156208 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:01.158850 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:01.162209 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:01.162644 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:01.175649 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:01.342679 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:03:01.397686 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:03:01.441123 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:01.473395 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:01.477909 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:01.479900 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:01.480808 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:01.481549 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:01.483649 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:01.490742 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:01.561084 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:03:01.797313 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:01.798449 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:01.800593 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:03:01.801833 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:01.803339 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:01.803748 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:01.813321 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:01.819860 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:01.908073 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:03:01.943482 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:01.955193 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:03:02.072023 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 02:03:02.123621 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:02.126771 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:02.128894 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:02.130807 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:02.141853 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:02.152168 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:02.153121 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:02.207528 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:03:02.325090 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 02:03:02.449081 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:02.455603 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:02.456063 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:02.456418 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:02.458792 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:02.461881 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:02.475751 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:02.495841 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:03:02.505422 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:02.777951 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:02.779950 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:02.780053 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:02.786201 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 02:03:02.790712 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:02.796967 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:02.803248 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:02.820359 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:03:02.830087 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:02.888455 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:03.114662 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:03.116895 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:03.120863 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:03.124116 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:03.125150 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:03.126502 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:03.128046 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:03.398808 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:03.442667 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:03.444897 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:03.445165 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:03.448581 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:03.449016 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:03.460661 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:03.479383 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:03.710417 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:03:03.725934 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:03:03.769773 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:03.769945 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:03.770795 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:03.777489 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:03.786131 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:03.786710 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:03.788980 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 02:03:03.813426 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:03.971400 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:04.025918 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:03:04.065518 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 02:03:04.103206 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:04.103507 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:04.111606 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:04.114604 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:04.114981 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:04.131572 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:04.148485 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:04.360028 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:04.394763 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:03:04.445330 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:04.447597 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:04.448793 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:04.450773 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:04.452584 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:04.454239 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:04.460155 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:04.497344 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:03:04.591789 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:03:04.601456 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:03:04.672225 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:03:04.766933 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:04.767292 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:04.773325 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:04.774539 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:04.775748 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:04.779398 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:04.804499 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:04.859086 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:03:04.859659 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:05.089205 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:05.094559 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:05.095667 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:05.103904 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:03:05.105372 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:05.105885 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:05.115024 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:05.118469 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:05.151404 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:03:05.290530 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:05.421231 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:05.421481 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:05.427003 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:05.427829 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:05.430553 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:05.439132 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:05.446216 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:05.538720 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:03:05.744943 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:05.746827 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:05.752372 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:05.752401 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:05.760159 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:05.775321 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:05.778699 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:05.827330 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:06.081122 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:06.083170 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:06.083699 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:06.088037 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:06.094553 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:06.100957 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:06.108959 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:06.268044 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:06.294809 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:03:06.295457 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:03:06.331452 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:03:06.414068 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:06.415258 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:06.415766 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:06.415805 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:06.416942 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:06.427793 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:06.429514 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:06.538904 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:03:06.687431 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:03:06.733702 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:06.735882 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:06.736366 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:06.737269 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:06.739132 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:06.740273 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:06.759207 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:06.853446 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:03:06.879094 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:06.890825 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:03:06.984010 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:03:07.051951 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:07.052548 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:03:07.060015 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:07.064534 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:07.065682 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:07.066352 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:07.074589 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:07.083914 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:07.229287 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:03:07.383323 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:03:07.384681 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:03:07.384768 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:07.393944 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:07.400568 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:07.410619 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:07.412964 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:07.415625 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:07.420895 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:07.431025 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:07.581933 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:03:07.724372 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:07.724690 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:07.725878 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:07.729480 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:07.730571 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:07.732648 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:07.736454 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:07.883814 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:08.046250 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:08.050484 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:08.050743 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:08.055303 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:08.058290 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:08.065168 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:08.073318 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:08.129940 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:03:08.365845 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:08.366379 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:08.372205 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:08.372963 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:08.375304 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:08.396579 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:08.397385 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:08.465826 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:08.608470 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:03:08.694926 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:08.702009 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:08.702255 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:08.702940 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:08.705751 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:08.722851 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:08.723222 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:08.859659 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:03:08.984812 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:09.014334 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:09.014575 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:09.014612 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:09.025269 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:09.027480 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:09.027905 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:03:09.036561 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:09.039917 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:09.094746 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:03:09.123298 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:03:09.283772 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:03:09.326059 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:03:09.336418 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:09.339109 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:09.342787 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:09.343660 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:09.345618 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:09.352509 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:09.359244 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:09.471609 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:03:09.494812 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:09.609205 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:03:09.672545 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:09.673535 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:09.676086 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:09.684355 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:09.686813 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:09.688503 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:09.693420 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:09.871659 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:03:09.880353 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:03:09.907956 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:03:09.984999 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:09.988677 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:09.990538 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:09.993808 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:09.995960 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:10.002569 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:10.004691 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:10.077994 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:10.190848 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:03:10.310833 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:10.311815 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:10.312093 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:10.326349 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:10.326737 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:10.334774 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:10.341923 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:10.631346 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:10.635175 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:10.635433 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:10.636096 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:10.644945 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:10.645080 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:10.651436 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:10.695941 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:10.804227 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:03:10.953677 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:10.960232 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:10.960680 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:10.966720 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:10.972509 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:10.982064 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:10.982239 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:11.079421 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:11.140895 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:03:11.276863 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:11.279150 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:11.282663 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:11.283896 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:11.301131 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:11.302000 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:11.310158 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:11.407352 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:03:11.466193 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:03:11.607802 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:11.613485 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:11.616935 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:11.617661 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:11.623347 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:11.623615 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:11.653479 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:11.657426 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:03:11.660557 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:11.661479 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:03:11.662204 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:03:11.747694 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:03:11.944012 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:11.947607 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:11.947894 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:11.950098 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:11.957795 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:11.965257 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:11.979496 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:12.042769 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:03:12.204375 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:12.216099 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:03:12.266319 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:12.275108 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:12.276242 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:12.276806 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:12.277516 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:12.278981 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:12.300849 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:12.370661 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:03:12.499743 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:03:12.569608 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:03:12.590590 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:12.591440 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:12.597776 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:12.607275 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:12.610576 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:12.611173 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:12.622026 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:12.753692 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:12.869846 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:03:12.913129 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:12.918198 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:12.919207 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:12.923450 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:12.923516 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:12.935629 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:12.958425 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:13.140873 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:13.219974 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:03:13.245494 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:13.247078 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:13.251135 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:13.251885 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:13.261576 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:13.261895 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:13.274642 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:13.451483 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:03:13.570081 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:13.572358 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:13.573431 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:13.581393 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:13.584348 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:13.591826 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:13.600786 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:13.613709 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:13.813410 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:03:13.820801 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:03:13.893357 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:13.894441 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:13.897202 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:13.897933 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:13.910485 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:13.911439 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:13.912214 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:14.038050 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:14.102819 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 02:03:14.213378 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:14.214143 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:14.214188 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:14.214751 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:14.221778 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 02:03:14.223357 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:14.224292 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:03:14.230597 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:14.237782 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:14.343554 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:03:14.520619 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:03:14.534443 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:14.536115 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:14.540639 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:14.543260 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:14.555161 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:14.557507 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:14.565521 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:14.617849 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:14.724749 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:03:14.851125 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 02:03:14.864995 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:14.869362 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:14.871695 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:14.875054 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:14.877262 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:14.887840 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:14.891510 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:14.911267 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:03:15.059976 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:03:15.060643 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:15.195158 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:15.204067 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:15.205392 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:15.205976 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:15.205976 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:15.206112 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:15.207270 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:15.406412 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:03:15.488451 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:15.519755 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:15.521183 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:15.521486 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:15.525771 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:15.531486 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:15.555563 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:15.557293 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:15.843817 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:15.845095 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:15.849416 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:15.859062 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:15.861591 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:15.872988 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:15.883546 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 02:03:15.888096 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:16.044263 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:03:16.074689 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:16.177048 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:16.177946 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:16.180637 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:16.182013 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:16.183475 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:16.192943 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:16.198598 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:16.338480 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:03:16.426049 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:03:16.497300 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 02:03:16.501076 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:16.502512 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:16.506686 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:16.510545 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:16.514515 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:16.514787 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:16.517160 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:16.546492 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:16.682101 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:03:16.682785 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:03:16.710945 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 02:03:16.823199 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:16.833432 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:16.835506 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:16.842492 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:16.843501 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:16.849740 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:16.863636 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:17.077173 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:17.112893 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 02:03:17.165406 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:17.170524 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:17.172806 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:17.173197 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:17.173997 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:17.182123 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:17.206372 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:17.215545 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:03:17.268257 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:03:17.473045 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:03:17.494614 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:17.506136 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:17.508562 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:17.508628 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:17.512616 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:17.517916 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:17.519099 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:17.530817 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:17.543711 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:03:17.816830 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:03:17.831130 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:17.831491 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:17.832733 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:17.834583 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:17.842588 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:17.849821 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:17.863338 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:17.931182 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:18.155751 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:18.160611 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:18.162493 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:18.162571 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:18.163439 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:18.166594 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:18.179455 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:18.302042 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:03:18.402271 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:18.469945 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:03:18.474673 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:18.474761 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:18.481909 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:18.498943 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:18.503186 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:18.503662 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:18.504060 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:18.808255 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:18.812652 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:18.815274 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:18.816881 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:18.823428 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:18.838006 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:18.842708 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:18.852289 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:03:18.909104 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:18.916934 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:03:19.036505 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:03:19.092491 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:03:19.136734 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:19.137043 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:19.137989 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:19.150242 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:19.150316 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:19.152216 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:19.177984 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:19.223749 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:03:19.225320 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:03:19.465734 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:19.471246 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:19.474556 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:19.478705 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:19.479683 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:19.499511 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:19.506842 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:19.523790 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:19.615026 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:03:19.627460 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:03:19.752792 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:03:19.799273 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:19.799780 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:19.809563 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:19.810787 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:19.815919 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:19.822204 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:19.833026 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:19.865011 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:03:20.000622 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:20.120743 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:20.126320 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:20.130131 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:20.130437 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:20.137692 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:20.150966 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:20.170912 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:20.353018 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:03:20.407243 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:03:20.458696 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:20.462424 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:20.462653 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:20.463749 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:20.479223 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:20.481317 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:20.492312 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:20.596853 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:20.778254 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:20.783227 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:20.783996 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:20.786216 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:20.802665 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:20.802829 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:20.804358 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:20.817938 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:03:20.861647 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:03:21.107239 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:21.111227 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:21.112176 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:21.116132 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:21.117859 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:21.118678 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:21.120023 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:21.149109 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:21.414047 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:03:21.437960 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:21.446362 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:21.450012 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:21.450582 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:21.451478 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:21.452159 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:21.453084 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:21.503616 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:03:21.523196 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:03:21.665979 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:03:21.707706 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:03:21.708397 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:03:21.767812 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:21.768914 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:21.771088 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:21.773315 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:21.775646 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:21.782424 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:21.782527 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:21.803536 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:22.081188 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:03:22.099117 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:22.100579 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:22.100666 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:22.101171 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:22.125218 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:22.129161 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:22.129951 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:22.173835 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:03:22.181710 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:22.203178 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:03:22.203272 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:03:22.429373 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:22.429518 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:22.434938 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:22.435156 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:22.438725 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:22.448220 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:22.456163 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:22.618098 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:03:22.673589 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:22.752290 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:22.752825 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:22.753274 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:22.771294 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:22.775393 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:22.775820 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:22.776103 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:22.779424 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:03:23.067492 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:23.074824 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:23.077654 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:23.083844 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:23.084321 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:23.091511 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:23.092413 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:23.110957 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:23.171667 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:03:23.259831 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:03:23.403426 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:23.404407 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:23.409446 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:23.418735 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:23.419349 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:23.420678 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:23.425988 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:23.658264 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:23.718652 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:23.723430 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:23.724825 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:23.726986 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:23.731699 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:23.734073 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:23.754547 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:23.862659 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:03:23.883031 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:03:23.940204 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:03:24.041505 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:24.047407 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:24.048281 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:24.055282 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:24.056236 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:24.066656 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:24.093686 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:24.159142 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:03:24.216383 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:24.348754 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:03:24.384676 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:24.385722 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:24.386042 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:24.394577 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:24.410702 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:24.413946 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:24.416855 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:24.437693 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:03:24.440925 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:03:24.441030 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:03:24.629109 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:03:24.681618 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:24.709905 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:24.710516 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:24.713266 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:24.723488 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:24.726924 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:24.731575 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:24.731873 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:24.752601 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:03:25.040194 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:25.046216 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:25.047214 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:25.047747 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:25.058179 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:25.071930 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:25.085454 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:25.240042 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:25.387051 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:25.393517 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:25.396886 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:25.409162 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:25.415005 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:25.420464 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:25.420959 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:25.471268 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:03:25.620211 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:03:25.620859 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:25.719784 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:25.721732 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:25.722669 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:25.723104 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:25.723430 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:25.724490 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:25.724628 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:25.803036 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:03:26.018875 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:03:26.035651 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:26.043037 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:26.045001 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:26.049150 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:26.049906 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:26.051483 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:26.059738 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:26.067134 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:26.333216 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:03:26.374629 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:26.378505 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:03:26.383884 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:26.386049 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:26.394667 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:26.396346 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:26.396975 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:26.410544 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:26.530216 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:03:26.554836 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:03:26.627208 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:26.642915 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:03:26.714102 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:26.714455 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:26.717985 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:26.718727 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:26.722118 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:26.730181 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:26.740875 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:26.888422 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:03:26.951148 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:03:26.960477 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:03:27.026438 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:03:27.048771 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:27.049089 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:27.051064 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:27.051585 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:27.053231 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:27.059147 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:27.067428 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:27.197067 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:27.331974 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:03:27.377506 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:27.378259 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:27.384145 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:27.385125 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:27.393569 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:27.408239 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:27.408510 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:27.631059 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:03:27.631882 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:03:27.708675 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:27.711184 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:27.714720 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:27.715271 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:27.718685 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:27.723977 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:27.728724 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:27.735764 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:28.024725 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:03:28.029136 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:28.043813 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:28.045861 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:28.053121 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:28.055028 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:28.055171 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:28.058281 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:28.208335 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:28.302421 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:03:28.367751 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:28.376073 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:28.382949 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:28.388414 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:28.389335 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:28.390045 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:28.411466 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:28.709877 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:28.715109 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:28.717425 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:28.718008 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:28.721007 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:28.721750 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:28.723233 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:28.757035 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:03:28.759498 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:03:28.812996 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:03:28.819218 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:29.017972 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:03:29.030033 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:29.035578 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:29.047058 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:29.047425 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:29.051330 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:29.056020 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:29.056455 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:29.363065 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:29.364816 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:03:29.365451 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:29.365725 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:29.371522 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:29.372181 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:29.406711 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:29.406994 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:29.408978 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:03:29.418831 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:29.542716 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:03:29.543287 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:03:29.631048 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:03:29.691254 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:29.693028 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:29.696033 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:29.700645 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:29.706689 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:29.710711 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:29.712507 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:30.015172 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:30.015563 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:30.016842 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:30.021962 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:30.027077 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:30.027981 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:30.042546 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:30.043851 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:30.079916 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:03:30.323046 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:03:30.324703 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:03:30.338088 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:30.339946 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:30.344532 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:30.348289 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:30.356950 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:30.356935 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:30.357290 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:30.459452 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:30.626317 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:03:30.669251 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:30.674013 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:30.674466 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:30.677218 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:30.682010 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:30.682494 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:30.691790 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:30.738618 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:03:30.974489 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:03:30.984943 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:30.988117 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:30.991067 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:30.991113 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:30.997341 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:31.000283 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:31.000459 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:31.067667 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:31.304196 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:31.307173 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:31.308573 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:31.309444 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:31.315973 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:31.320710 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:31.321630 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:31.356312 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:03:31.490581 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:03:31.606292 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:31.631536 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:31.636138 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:31.638646 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:31.644793 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:31.651448 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:31.654276 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:31.672390 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:31.688663 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:03:31.767484 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:03:31.791514 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:03:31.821897 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:03:31.962692 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:31.965977 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:31.970609 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:31.971690 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:31.978576 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:31.983979 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:31.990597 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:32.033077 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:03:32.036602 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:03:32.075174 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:32.283046 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:32.287201 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:32.288317 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:32.288934 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:32.294728 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:32.302644 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:32.316550 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:32.408766 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:03:32.604956 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:32.605792 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:32.608000 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:32.610502 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:32.616396 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:32.620369 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:32.628080 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:32.646807 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:32.654553 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:03:32.680748 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:03:32.927806 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:32.930006 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:32.932672 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:32.933501 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:32.934422 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:32.937964 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:32.943423 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:33.088375 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:33.131559 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:03:33.157727 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:03:33.251569 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:33.252591 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:33.253281 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:33.258899 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:33.262732 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:33.273139 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:33.286844 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:33.493592 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:33.563413 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:03:33.573215 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:33.574347 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:33.576545 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:33.577215 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:33.584779 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:33.594233 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:33.612908 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:33.877380 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:33.894604 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:03:33.900599 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:33.901079 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:33.905853 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:33.911863 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:33.916779 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:33.933264 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:33.933992 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:34.034978 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:03:34.077381 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:03:34.224488 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:34.226040 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:34.226754 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:34.227488 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:34.232676 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:34.234658 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:34.244577 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:34.261720 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:03:34.332612 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:03:34.406107 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:03:34.453842 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:03:34.494928 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:34.548889 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:34.549489 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:34.552526 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:34.558240 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:34.562670 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:34.576534 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:34.579200 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:34.766693 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:03:34.856920 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 02:03:34.872471 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:34.878690 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:34.879379 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:34.879848 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:34.880347 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:34.883538 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:34.907943 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:34.916006 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:35.193238 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:03:35.204222 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:35.204522 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:35.208346 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:35.218145 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:35.221102 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:35.221876 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:35.227490 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:35.329416 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:35.530452 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:35.536015 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:35.536537 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:35.539227 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:35.540664 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:35.560221 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:35.562031 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:35.617847 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:03:35.665061 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:03:35.760834 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:03:35.849565 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:03:35.850355 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:35.862751 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:35.863296 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:35.863324 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:35.864193 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:35.869421 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:35.870359 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:35.882827 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:36.187492 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:36.194991 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:36.199405 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:36.199774 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:36.199806 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:36.200388 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:36.206811 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:36.300276 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:03:36.401763 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:36.517444 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:36.522516 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:36.526217 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:36.527172 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:36.528332 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:36.534731 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:36.540606 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:36.729925 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:03:36.839133 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:36.839577 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:36.842115 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:36.849862 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:36.851011 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:36.854178 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:36.858191 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:36.876452 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:36.890332 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:03:36.908128 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:03:36.924535 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:03:36.996218 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:03:37.043853 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:03:37.168196 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:37.172065 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:37.172576 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:37.178438 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:37.182127 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:37.195449 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:37.206977 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:37.218832 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:03:37.374812 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:03:37.417949 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:37.502172 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:37.503615 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:37.505154 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:37.508105 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:37.510571 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:37.518761 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:37.527230 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:37.702248 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:03:37.708166 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240502 02:03:37.822057 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:37.824724 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:37.825917 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:37.827251 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:37.828333 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:37.839828 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:37.861469 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:37.911146 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:38.147115 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:38.150710 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:38.153649 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:38.160745 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:38.161419 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:38.177798 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:38.182382 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:38.312721 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:03:38.453217 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:38.471383 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:03:38.480139 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:38.482175 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:38.484556 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:38.485756 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:38.488626 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:38.498500 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:03:38.512503 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:38.514094 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:38.734025 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:03:38.810716 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:38.812178 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:38.814778 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:38.822757 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:38.829020 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:38.841351 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:38.845384 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:38.862663 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:39.141438 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:39.141489 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:39.145689 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:39.146229 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:39.156714 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:39.167503 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:39.180074 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:39.216398 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:03:39.331984 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:03:39.347707 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:39.415138 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:03:39.465308 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:39.467234 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:39.469187 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:39.470152 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:39.470729 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:39.487682 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:39.490360 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:03:39.506669 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:39.611564 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:03:39.625202 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:03:39.793521 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:39.803267 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:39.807101 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:39.808411 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:39.808687 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:39.821889 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:39.838003 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:39.890071 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:03:39.902151 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:40.015655 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:03:40.125064 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:40.125559 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:40.128719 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:40.128981 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:40.146561 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:40.151669 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:40.152082 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:40.350529 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:03:40.454530 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:40.455539 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:40.461634 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:40.467978 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:40.468940 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:40.470541 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:40.483003 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:40.485934 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:40.783565 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:40.785221 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:40.786808 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:40.789569 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:40.798472 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:40.804960 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:40.821646 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:40.836987 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:03:40.873152 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:03:40.988418 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:03:41.013345 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:03:41.014045 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:41.108879 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:41.110337 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:41.113386 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:41.116449 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:41.127748 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:41.134920 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:41.146937 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:41.427968 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:03:41.447345 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:41.447361 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:41.448957 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:41.450912 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:41.461932 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:41.462627 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:41.471372 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:41.616640 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:41.727435 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:03:41.767343 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:41.770836 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:41.776170 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:41.777921 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:41.779405 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:41.790853 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:41.791103 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:42.050791 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:03:42.105621 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:42.111164 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:42.111394 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:42.116737 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:42.116930 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:03:42.117628 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:42.121294 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:42.121982 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:42.132652 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:03:42.147327 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:03:42.178597 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:03:42.198725 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:42.341079 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:03:42.386484 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:03:42.446961 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:42.448868 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:42.449457 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:42.457870 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:42.460496 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:42.461088 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:42.483716 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:42.606331 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:42.726588 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:03:42.750360 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:03:42.778488 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:42.782197 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:42.783344 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:42.785729 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:42.785871 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:42.794590 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:42.801081 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:43.073202 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:43.111976 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:43.116639 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:43.118036 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:43.121922 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:43.125099 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:43.132866 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:43.161474 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:43.349157 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:03:43.452908 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:43.458130 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:43.458230 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:43.459080 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:43.470232 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:43.471441 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:03:43.471865 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:43.494206 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:43.580816 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:43.691944 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:03:43.762701 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:03:43.787079 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:43.789299 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:43.789764 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:43.792503 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:43.793077 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:43.794559 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:43.821910 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:43.993162 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:44.088032 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:03:44.120108 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:44.127789 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:44.131536 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:44.135938 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:44.135990 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:44.153053 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:44.160593 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:44.456725 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:44.462321 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:44.466682 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:44.466806 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:44.483106 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:44.485867 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:44.495525 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:44.517247 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:03:44.538698 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:03:44.556825 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:44.613343 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:03:44.760147 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:03:44.785843 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:44.786892 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:44.789990 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:44.790500 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:44.802582 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:44.807368 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:44.809070 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:44.842180 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:03:44.893980 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 02:03:45.007184 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:45.110507 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:45.116397 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:45.116591 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:45.117363 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:45.127666 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:45.135044 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:45.136014 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:45.191826 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:03:45.460201 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:45.464803 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:45.465298 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:45.468617 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:45.468628 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:03:45.475181 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:45.478758 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:45.483645 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:45.486384 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:45.790820 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:45.792886 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:45.795127 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:45.796027 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:45.799191 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:45.815904 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:45.821924 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:46.003382 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:03:46.040323 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:03:46.040944 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:46.055419 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:03:46.114089 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:46.116001 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:46.117651 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:46.119325 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:46.121366 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:46.135406 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:46.138018 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:46.225051 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:03:46.339825 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:03:46.449190 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:46.450047 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:46.450846 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:46.451027 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:46.457572 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:46.481252 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:46.483417 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:46.570665 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:46.720096 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:03:46.772346 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:46.774097 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:46.775024 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:46.781816 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:46.782166 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:46.782953 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:46.795648 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:46.906929 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:03:46.940791 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:03:46.994851 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:03:47.094957 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:47.100072 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:47.101596 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:47.103698 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:47.104441 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:47.120816 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:47.126585 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:47.130657 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:47.167805 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:03:47.208707 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:03:47.395699 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 02:03:47.435972 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:47.436832 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:47.440441 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:47.445293 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:47.446722 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:47.452951 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:47.457702 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:47.526242 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:47.726420 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:03:47.752657 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:47.759071 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:47.761884 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:47.762614 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:47.762806 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:47.770550 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:47.789692 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:47.819651 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:03:47.824069 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 02:03:48.078542 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:48.079141 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:48.088600 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:48.090628 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:48.090795 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:48.092053 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:48.099977 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:48.107856 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:48.317561 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:03:48.338093 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:03:48.414820 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:48.416147 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:48.417704 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:48.420410 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:48.428818 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:48.434562 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:48.436882 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:48.485757 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:48.676213 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:03:48.737684 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:48.742944 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:48.743834 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:48.744427 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:48.746531 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:48.747220 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:48.748536 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:48.949846 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:48.987926 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:03:49.061741 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:49.063207 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:49.068008 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:49.070166 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:49.070813 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:49.078505 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:49.097827 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:49.124361 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:03:49.393860 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:49.406680 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:49.408134 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:49.413527 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:49.414050 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:49.416617 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:49.429980 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:49.439853 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:49.446492 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:03:49.468333 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:03:49.546801 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:03:49.633878 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:03:49.721271 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:49.722537 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:49.723592 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:49.735090 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:49.735738 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:49.738976 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:49.751601 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:49.825769 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:03:49.965157 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:50.043329 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:50.043992 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 02:03:50.045855 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:50.048663 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:50.057089 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:50.062201 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:50.063591 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:03:50.071220 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:50.074633 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:50.194878 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:03:50.371381 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:50.373225 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:50.375135 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:50.378749 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:50.392103 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:50.392760 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:50.397280 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:50.432214 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:50.709142 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:50.714807 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:50.715960 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:50.716671 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:50.723556 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:50.725553 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:50.725833 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:50.819192 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:03:50.934898 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:51.031327 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:51.032003 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:51.044842 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:51.045336 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:51.046780 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:51.054821 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:51.059744 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:51.158789 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:03:51.163789 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:51.299822 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:03:51.349040 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:51.352751 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:51.358433 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:51.362720 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:51.362879 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:51.363452 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:51.363683 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:51.403573 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:51.618386 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:03:51.695569 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:51.696036 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:51.697374 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:51.705302 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:51.710453 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:51.728103 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:51.730676 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:51.801285 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:51.814280 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:03:51.836313 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:03:51.933686 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:03:52.017671 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:52.019740 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:52.026962 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:52.029605 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:52.030738 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:03:52.038973 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:52.053182 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:52.064361 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:52.327273 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:03:52.350605 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:52.352499 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:52.354717 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:52.355748 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:52.359310 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:52.361701 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:52.362845 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:03:52.381448 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:52.410322 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:52.459372 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:03:52.460103 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:03:52.688720 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:52.689677 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:52.689798 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:52.690713 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:52.698706 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:52.707461 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:52.713798 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:52.730026 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:03:52.861063 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:03:52.918641 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:53.003770 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:53.004812 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:53.010531 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:53.014001 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:53.027475 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:53.032303 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:53.039031 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:53.333339 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:53.333436 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:53.337868 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:53.338302 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:53.341229 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:53.343626 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:53.344910 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:53.418998 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:53.525275 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:53.654497 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:03:53.660622 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:53.661970 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:53.666383 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:53.674460 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:53.675691 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:53.679090 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:53.679337 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:53.732277 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:53.956298 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:53.983219 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:53.986295 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:53.987542 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:53.990332 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:53.995857 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:54.006115 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:54.021697 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:54.032909 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:54.282869 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:54.302139 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:03:54.306046 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:54.306046 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:54.312917 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:54.313369 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:54.326566 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:54.330960 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:54.350603 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:54.451222 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:54.552549 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:03:54.633870 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:54.638968 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:54.640812 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:54.646560 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:54.654619 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:54.656755 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:54.666196 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:54.677075 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:54.836990 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:03:54.958245 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:54.963745 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:54.963961 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:54.968986 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:54.976910 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:54.979264 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:54.987712 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:55.033658 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:55.066901 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:55.093554 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:03:55.098292 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:03:55.113471 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:03:55.282984 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:55.288120 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:55.291954 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:55.292834 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:55.293422 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:55.303258 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:55.307866 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:55.608239 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:55.608695 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:55.610304 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:55.619294 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:55.620471 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:55.620963 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:55.628407 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:55.655395 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:55.866050 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:03:55.933147 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:55.933629 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:55.939554 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:55.942114 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:55.942736 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:55.953289 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:55.972518 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:56.000743 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:56.059456 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:56.197527 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:03:56.252688 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:56.258165 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:56.258188 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:56.259203 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:56.262151 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:56.271484 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:56.283931 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:56.425292 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:56.524920 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:03:56.525578 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:56.575968 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:56.580089 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:56.582671 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:56.583505 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:56.586719 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:56.587006 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:56.593492 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:56.818938 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:56.867579 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:03:56.896483 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:56.897821 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:56.898633 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:56.901544 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:56.904982 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:56.909154 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:56.909612 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:56.929720 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:56.995931 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:03:57.215993 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:03:57.225319 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:57.226743 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:57.227809 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:57.238619 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:57.246593 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:57.248001 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:57.259652 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:57.309899 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:03:57.364280 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:57.417804 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:57.540359 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:03:57.553676 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:03:57.555047 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:03:57.558059 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:57.561789 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:57.563740 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:57.564211 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:57.567410 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:57.577523 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:03:57.845839 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:57.881490 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:57.882310 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:57.883116 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:57.885974 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:57.888437 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 02:03:57.894266 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:57.896288 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:57.913813 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:57.960736 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:03:58.183400 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:03:58.210721 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:58.211855 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:58.213445 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:58.216327 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:58.225837 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:58.231256 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:58.243052 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:58.444182 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:03:58.471549 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:03:58.542384 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:58.545955 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:58.548012 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:58.550585 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:58.551915 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:58.554589 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:58.568749 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:58.764907 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:03:58.870322 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:58.874070 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:58.879880 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:58.888799 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:58.889209 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:58.890931 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:58.901321 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:59.028810 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:03:59.031486 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:59.058241 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:59.201005 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:59.205114 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:59.206715 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:59.209645 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:03:59.209766 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:59.209962 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:59.214609 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:03:59.389501 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:59.474596 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:03:59.526010 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:59.532969 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:59.533244 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:59.536255 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:59.538384 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:03:59.538385 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:59.563381 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:59.689458 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:03:59.774261 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:03:59.853119 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:59.853494 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:59.854113 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:03:59.856940 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:03:59.859931 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:03:59.873320 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:03:59.874692 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:03:59.876792 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:03:59.951668 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:04:00.036777 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:00.076627 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:04:00.178820 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:00.182546 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:00.182919 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:00.188308 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:00.190140 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:00.199180 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:00.211448 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:00.220885 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:04:00.238160 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:04:00.514119 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:00.515174 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:00.515463 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:00.516546 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:00.517779 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:00.519973 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:00.529242 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:00.602631 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:04:00.644991 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:00.840814 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:00.842555 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:00.842620 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:00.844923 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:00.854947 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:00.861534 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:00.861595 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:01.172786 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:01.174593 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:01.178546 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:01.181478 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:01.182363 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:01.184707 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:01.195839 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:04:01.205569 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:01.257526 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:01.499595 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:01.504201 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:01.509238 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:01.512006 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:01.515367 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:01.523152 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:01.526279 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:01.574641 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 02:04:01.656194 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:04:01.670042 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:01.753319 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:04:01.823575 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:01.824069 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:01.830694 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:01.835606 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:01.842216 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:01.844081 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:01.850591 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:01.951169 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:04:02.055840 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:02.113790 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:04:02.153833 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:02.157027 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:02.163903 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:02.164546 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:02.167559 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:02.168754 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:02.175300 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:02.289239 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:04:02.289955 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:04:02.331631 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:04:02.459497 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:04:02.481623 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:02.484019 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:02.491515 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:02.491708 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:02.502703 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:02.502923 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:02.504241 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:02.505089 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:02.651861 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:04:02.688669 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:04:02.813948 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:02.816195 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:02.817414 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:02.822278 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:02.822259 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:02.822913 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:02.839947 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:02.897997 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240502 02:04:03.080842 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:03.133307 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:03.134450 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:03.139786 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:03.142586 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:03.145000 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:03.155112 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:03.157132 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:03.201876 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:04:03.279595 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:04:03.458910 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:03.463690 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:03.464119 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:03.466344 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:03.467862 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:03.484653 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:03.495321 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:03.495393 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:03.667347 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:04:03.782384 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:03.783885 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:03.789104 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:03.802664 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:03.802760 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:03.804728 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:03.806032 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:04.069979 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:04.112340 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:04.115692 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:04.121830 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:04.122005 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:04.127346 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:04.144320 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:04:04.145762 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:04.150700 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:04.313616 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:04:04.437893 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:04.438259 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:04.439622 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:04.444407 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:04.446784 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:04.447515 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:04.447898 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:04.551798 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:04.695659 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:04:04.716930 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:04:04.749430 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:04.752259 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:04.756345 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:04.758553 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:04.769413 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:04.770846 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:04.771427 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:04.782333 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:04:04.801309 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:04:05.065068 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:05.070920 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:05.075781 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:05.076092 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:04:05.078617 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:05.081212 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:05.081219 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:05.082367 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:05.090627 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:05.289383 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:04:05.299525 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 02:04:05.300206 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:04:05.380949 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:04:05.404383 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:05.406281 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:05.409224 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:05.414554 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:05.419044 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:05.424073 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:05.431150 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:05.583747 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:05.726018 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:05.728713 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:05.731015 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:05.733850 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:05.734421 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:05.742553 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:05.746560 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:06.015410 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:06.040496 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:04:06.046617 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:04:06.053067 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:06.054124 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:06.055397 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:06.065448 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:06.067772 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:06.079411 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:06.080410 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:06.305984 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:04:06.384847 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:06.386147 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:06.389406 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:06.396369 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:06.397686 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:06.414816 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:06.415624 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:06.471320 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:06.720188 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:06.720813 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:06.722708 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:06.723145 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:06.730716 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:06.732473 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:06.733911 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:06.798194 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:04:06.874307 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:04:06.997063 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:07.033684 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:07.041921 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:07.045631 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:07.046342 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:07.058540 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:07.058718 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:07.083045 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:07.155036 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:04:07.158751 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:04:07.295959 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:04:07.369699 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:07.375062 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:07.377622 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:07.392117 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:07.392146 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:07.392632 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:07.412142 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:07.497088 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:04:07.561260 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:07.590286 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:04:07.708312 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:07.709142 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:07.709897 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:07.711020 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:07.716431 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:07.717767 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:07.728317 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:07.896523 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:04:07.909238 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:04:07.981508 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:04:08.028416 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:08.031039 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:08.045804 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:08.046152 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:08.047750 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:08.048007 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:08.048483 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:08.114570 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:04:08.128095 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:08.281116 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:04:08.366230 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:08.366539 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:08.367280 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:08.370787 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:08.370924 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:08.372256 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:08.402855 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:08.526786 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:08.544598 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 02:04:08.693027 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:08.694833 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:08.699324 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:08.705454 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:08.705612 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:08.706243 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:08.714912 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:04:08.721369 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:08.899379 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:04:08.967798 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:09.021078 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:09.024050 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:09.024992 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:09.025385 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:09.025689 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:09.029472 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:09.058491 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:09.354465 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:09.356837 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:09.356837 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:09.363801 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:09.368170 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:09.372097 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:09.373510 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:09.508214 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:04:09.521632 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:04:09.547238 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:09.643985 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:04:09.675182 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:04:09.688045 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:09.693212 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:09.694583 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:09.696218 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:09.696444 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:09.698128 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:09.713600 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:09.925447 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:10.005990 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:10.011289 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:10.012334 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:10.019176 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:10.023198 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:10.025481 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:10.029115 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:10.125131 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:04:10.131763 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:04:10.337810 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:10.340350 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:10.342674 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:10.347366 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:10.356254 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:10.362709 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:10.376499 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:10.431213 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:10.439294 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 02:04:10.439927 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:04:10.540357 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:04:10.578565 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:04:10.678344 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:10.680193 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:10.682018 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:10.685624 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:10.687069 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:10.696734 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:10.697218 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:04:10.706593 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:10.910120 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:11.005456 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:11.006309 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:11.012795 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:11.020355 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:11.022820 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:11.023564 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:11.042086 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:11.140018 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:04:11.340042 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:11.343258 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:11.344408 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:11.345783 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:11.352221 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:11.368842 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:11.372344 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:11.484556 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:11.542860 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:04:11.692080 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:11.698264 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:11.698635 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:11.699275 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:11.702555 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:11.705332 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:11.718895 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:11.880584 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:04:11.881245 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:11.931404 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:04:11.947235 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:04:12.018683 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:12.023146 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:12.023437 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:12.039021 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:12.043037 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:12.043751 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:12.065212 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:12.192274 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:04:12.244994 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:04:12.359473 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:12.360263 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:12.360291 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:12.360450 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:12.367908 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:12.377398 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:12.377841 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:12.379884 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:12.494580 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:04:12.637528 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:04:12.691532 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:12.696079 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:12.699122 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:12.702605 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:12.706939 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:12.707067 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:12.720746 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:12.879827 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:04:12.955812 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:12.982414 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:04:13.023360 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:13.026438 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:13.026785 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:13.029753 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:13.038578 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:13.055212 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:13.056133 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:13.094208 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:04:13.345559 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:13.352105 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:13.357587 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:13.357637 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:13.362265 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:13.362579 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:13.365522 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:13.368556 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:13.488302 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:04:13.489046 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:04:13.620811 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:04:13.621672 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:04:13.686775 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:13.692837 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:13.694561 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:13.696254 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:13.696293 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:13.712327 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:13.715158 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:13.892304 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:14.003422 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:14.010470 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:14.010927 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:14.012290 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:14.012943 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:14.021867 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:14.023017 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:04:14.053877 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:14.336083 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:14.336591 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:14.343278 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:14.346539 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:14.351604 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:14.359532 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:14.377638 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:14.436203 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:04:14.478830 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:14.606202 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:04:14.618443 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:04:14.661506 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:04:14.672328 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:14.673336 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:14.675091 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:14.676286 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:14.683765 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:14.686646 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:14.687798 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:14.888253 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:14.988306 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:14.991647 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:14.992384 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:14.993327 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:14.997982 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:15.005403 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:15.018090 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:15.061635 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:04:15.174264 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:04:15.293190 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:04:15.315104 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:15.321159 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:15.322580 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:15.323993 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:15.324251 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:15.327584 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:15.337141 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:15.375486 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:04:15.428279 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:04:15.481177 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:15.636814 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:15.639633 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:15.642725 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:15.645423 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:15.646579 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:15.659437 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:15.670204 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:15.783780 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:04:15.856505 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:04:15.947188 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:04:15.972651 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:15.972987 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:15.976128 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:15.978616 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:15.978714 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:15.992636 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:15.999013 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:16.006639 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:16.293591 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:16.293972 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:16.294529 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:16.297892 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:16.301924 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:16.314577 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:16.316257 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:16.390794 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:16.548533 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:04:16.627164 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:16.629864 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:16.629998 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:16.631793 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:16.635933 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:16.636219 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:16.656960 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:16.801293 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:16.954878 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:16.955219 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:16.956107 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:16.958261 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:16.961120 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:16.974155 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:16.974556 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:17.083993 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:04:17.162959 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:04:17.213065 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:04:17.216147 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:04:17.229738 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:17.282097 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:17.282548 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:17.284477 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:17.299010 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:17.299670 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:17.311405 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:17.314581 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:17.321148 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:04:17.498579 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:04:17.608670 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:17.613216 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:17.614529 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:17.620201 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:17.623613 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:17.623939 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:17.638268 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:17.754261 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:17.763311 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:04:17.779129 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:04:17.901588 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:04:17.903975 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:04:17.937366 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:17.937568 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:17.939635 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:17.943382 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:17.951560 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:17.956364 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:17.969725 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:18.263458 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:18.265810 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:18.266819 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:18.266840 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:18.268569 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:18.269731 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:18.283008 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:18.283795 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:18.304533 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:04:18.367972 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:04:18.564253 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 02:04:18.594645 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:18.595692 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:18.597611 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:18.598031 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:18.602185 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:18.631126 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:18.631186 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:18.746443 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:04:18.759902 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:18.926374 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:18.927100 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:18.927448 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:18.928143 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:18.932255 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:18.949147 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:18.949798 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:19.186000 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:04:19.237144 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:19.251490 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:19.257055 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:19.261536 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:19.262480 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:19.269156 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:19.272274 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:19.276062 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:19.586681 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:19.588438 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:19.590631 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:19.598564 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:19.606099 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:19.607262 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:04:19.615264 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:19.618050 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:19.652196 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:19.703473 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:04:19.850973 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:04:19.871861 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:04:19.916020 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:19.923974 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:19.924762 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:19.931663 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:19.932482 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:19.933380 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:19.934773 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:20.129319 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:04:20.157248 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:20.191529 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:04:20.248875 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:20.252357 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:20.253757 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:20.260321 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:20.262787 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:20.269883 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:20.287153 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:20.303225 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:04:20.305783 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:04:20.392266 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:04:20.579767 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:20.582525 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:20.586251 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:20.590145 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:20.610303 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:20.610579 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:20.613226 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:20.618873 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:20.864655 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:04:20.892047 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:04:20.902284 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:20.903237 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:20.913789 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:20.917044 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:20.917680 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:20.934607 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:20.936806 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:21.097066 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:21.185763 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240502 02:04:21.230791 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:21.234287 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:21.236352 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:21.242599 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:21.242808 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:21.245801 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:21.247846 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:21.446944 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:04:21.483448 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:21.556898 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:21.564214 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:21.564908 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:21.567744 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:21.567901 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:21.571485 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:21.598906 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:21.888795 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:21.891710 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:21.891827 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:21.893546 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:21.897408 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:21.897790 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:21.914366 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:21.983320 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:22.067247 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:04:22.199542 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:04:22.209996 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:22.218968 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:22.221675 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:22.224086 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:22.229758 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:22.231673 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:22.237743 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:04:22.238567 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:22.516534 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:04:22.544723 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:22.546626 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:22.547044 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:22.547251 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:22.549357 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:22.552892 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:22.561851 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:22.573830 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:04:22.582840 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:04:22.605430 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:04:22.606110 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:22.687152 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:04:22.732980 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:04:22.867010 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:22.867275 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:22.873306 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:22.876952 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:22.877307 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:22.882975 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:22.887295 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:23.033347 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:04:23.128775 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:23.194218 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:23.197140 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:23.202538 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:23.206418 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:23.207230 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:23.235726 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:23.235715 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:23.316246 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:04:23.362020 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:04:23.526225 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:23.528131 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:23.529943 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:23.536585 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:23.544386 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:23.547572 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:23.551129 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:23.558928 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:23.571801 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:04:23.777735 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:04:23.851178 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:23.856585 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:23.857396 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:23.857658 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:23.859823 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:23.869971 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:23.880832 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:23.917775 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:04:24.172266 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:24.184453 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:24.185809 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:24.186241 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:24.188920 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:24.197620 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:24.207691 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:24.207998 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:24.489022 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:04:24.504554 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:24.511545 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:24.511956 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:24.514523 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:24.515142 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:24.515432 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:24.526803 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:24.566989 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:04:24.709470 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:04:24.791285 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:24.831269 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:24.832222 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:24.840833 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:24.841109 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:24.841384 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:24.842914 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:24.862007 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:25.042274 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:04:25.082744 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:04:25.094887 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:04:25.153491 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:25.156806 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:25.161072 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:25.162865 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:25.163028 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:25.177506 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:25.179272 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:25.186352 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:25.213361 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:04:25.312232 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:04:25.471748 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:04:25.479353 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:25.479565 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:25.479849 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:25.482815 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:25.486557 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:25.501245 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:25.513782 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:25.695185 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:04:25.703336 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:25.799448 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:25.801723 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:25.802058 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:25.802318 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:25.813959 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:25.814742 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:25.837976 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:26.003885 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:04:26.122965 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:26.124261 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:26.124850 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:26.136227 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:26.138242 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:26.151752 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:26.153661 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:26.208505 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 02:04:26.221956 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:26.450044 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:26.451265 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:26.455374 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:26.459869 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:26.461786 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:26.473114 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:26.478847 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:26.665040 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:04:26.665150 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:26.769136 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:04:26.774153 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:26.775974 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:26.777401 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:26.777433 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:26.783147 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:26.799681 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:26.813174 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:26.948032 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:04:27.099563 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:27.103785 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:27.106102 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:27.110822 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:27.112591 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:27.126225 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:27.128953 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:27.148676 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:27.318619 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:04:27.342263 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:04:27.356194 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:04:27.431732 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:27.434163 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:27.434479 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:27.449631 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:27.451035 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:27.457428 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:27.457547 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:27.576908 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:04:27.638846 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:04:27.639432 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:27.744992 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:27.746958 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:27.751055 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:27.760068 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:27.761405 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:27.762552 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:27.765558 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:27.770447 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:04:27.795696 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:04:27.922960 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:04:28.041261 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:28.066299 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:28.066351 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:28.071640 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:28.071975 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:28.082336 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:28.091377 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:28.105731 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:28.166584 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:04:28.390242 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:28.397181 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:28.408783 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:28.409484 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:28.411229 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:28.412245 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:28.414744 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:28.437741 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:04:28.448148 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:28.576679 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 02:04:28.721992 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:28.723850 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:28.724241 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:28.725239 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:28.727303 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:28.734555 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:28.749642 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:28.829681 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:04:29.039041 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:29.046063 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:29.051582 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:29.054330 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:29.059873 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:29.066967 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:29.071094 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:29.074977 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:29.111416 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:04:29.187135 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:04:29.352553 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:04:29.375533 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:29.376739 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:29.390573 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:29.391108 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:29.396297 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:29.417510 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:29.423660 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:29.657967 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:29.710731 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:04:29.714321 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:29.715497 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:29.718566 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:29.720382 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:29.727981 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:29.732745 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:29.753108 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:29.798888 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:04:29.822978 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:04:30.045833 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:30.054733 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:30.055728 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:30.055963 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:30.057368 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:30.059774 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:30.071563 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:30.079172 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:30.287652 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:04:30.355593 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:04:30.371882 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:30.378881 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:30.379063 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:30.383464 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:30.391539 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:30.398383 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:30.418956 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:30.474416 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:04:30.483865 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:30.515051 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:04:30.705605 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:30.711853 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:30.712061 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:30.726794 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:30.727272 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:30.729378 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:30.739220 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:30.807516 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:04:30.946615 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:04:31.024587 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:31.030910 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:31.031484 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:31.033162 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:31.046662 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:31.046892 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:31.066243 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:31.082828 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:31.273070 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 02:04:31.355464 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:31.358232 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:31.362051 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:31.362653 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:31.373907 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:31.378731 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:31.397023 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:31.429670 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:04:31.552886 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:31.685822 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:31.692462 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:31.693684 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:31.695482 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:31.705224 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:31.707681 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:31.721772 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:31.851734 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:04:32.002991 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:04:32.014031 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:32.018484 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:32.020341 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:32.022564 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:32.027457 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:32.033128 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:32.057714 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:04:32.065290 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:32.103024 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:04:32.110064 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:32.307951 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:04:32.352722 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:32.356142 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:32.365456 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:32.365736 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:32.366027 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:32.368711 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:32.369210 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:32.498212 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:04:32.685608 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:32.689579 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:32.695190 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:32.695478 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:32.704171 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:32.708864 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:32.727607 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:04:32.728529 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:32.729535 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:32.845427 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:04:32.925521 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:04:33.015378 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:33.016429 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:33.018973 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:33.025930 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:33.026635 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:33.035909 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:04:33.037503 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:33.044132 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:33.165233 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:33.335220 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:04:33.342624 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:33.344723 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:33.345168 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:33.350730 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:33.358569 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:33.363457 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:33.366716 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:33.553983 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:04:33.578900 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 02:04:33.682003 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:33.682317 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:33.684880 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:33.695784 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:33.695806 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:33.698628 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:33.698956 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:33.778795 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:33.909317 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:04:34.003249 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:34.006831 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:34.007607 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:34.010355 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:34.025667 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:34.026994 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:34.036036 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:34.100028 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:04:34.306989 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:34.335906 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:34.337985 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:34.338274 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:34.346982 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:34.349574 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:34.350783 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:34.355404 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:34.409658 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:04:34.432109 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:04:34.459286 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:04:34.628762 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:04:34.667642 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:34.669772 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:34.672907 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:34.677057 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:34.678349 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:34.688721 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:34.695551 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:34.774094 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:34.824993 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:04:34.979722 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:34.986524 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:34.986529 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:34.988550 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:34.989744 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:34.991694 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:35.016134 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:35.156936 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:04:35.302561 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:35.304064 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:35.304345 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:35.304520 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:35.307078 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:35.308007 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:35.329632 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:35.338909 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:35.447126 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:04:35.453843 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:04:35.598213 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:04:35.628686 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:35.629088 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:35.629535 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:35.638407 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:35.650357 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:35.654273 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:35.659255 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:35.700779 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:04:35.950112 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:35.951193 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:35.954423 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:35.955281 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:35.956682 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:35.959197 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:35.966851 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:35.966987 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:36.257033 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:04:36.268949 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:36.272680 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:36.275852 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:36.276273 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:36.278367 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:36.292887 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:36.303385 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:36.389600 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 02:04:36.522500 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:36.594084 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:36.594669 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:36.594758 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:36.596626 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:36.609591 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:36.617264 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:36.620191 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:36.676868 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:04:36.891232 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:04:36.918645 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:36.925658 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:36.927418 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:36.927526 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:36.928131 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:36.931540 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:36.940286 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:37.038529 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:04:37.046435 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:37.048547 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:04:37.126444 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:04:37.243686 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:37.245682 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:37.247085 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:37.250700 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:37.251541 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:37.267066 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:37.270265 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:37.324208 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:04:37.564184 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:37.569118 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:37.570463 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:37.574227 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:37.575259 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:37.588147 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:37.588142 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:37.626837 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:37.858861 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:04:37.890674 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:04:37.891827 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:04:37.902043 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:37.904053 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:37.907857 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:37.915156 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:37.917981 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:37.923079 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:37.935235 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:37.984720 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:04:38.096750 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:38.190773 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:04:38.220078 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:38.222491 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:38.226240 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:38.228916 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:38.237918 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:38.238160 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:04:38.238628 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:38.239013 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:38.496892 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:38.546150 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:38.546981 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:38.548573 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:38.549674 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:38.555228 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:38.562885 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:38.578002 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:38.609755 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:04:38.806042 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:04:38.870584 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:38.874246 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:38.875026 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:38.877758 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:38.889504 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:38.891331 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:38.908257 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:39.009763 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:39.068032 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:04:39.199872 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:39.202811 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:39.203413 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:04:39.207167 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:04:39.208609 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:39.209830 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:39.210634 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:39.220906 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:39.222920 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:39.436305 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:04:39.525465 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:39.525599 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:39.530372 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:39.538794 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:39.544200 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:39.554018 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:39.563102 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:39.563165 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:04:39.608428 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:39.684651 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:04:39.853497 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:39.858661 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:39.867213 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:39.867547 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:39.869740 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:39.870565 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:39.882227 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:39.992086 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:04:40.007915 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:40.183092 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:40.184599 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:40.189558 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:40.189738 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:40.195858 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:40.199851 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:40.209836 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:04:40.217648 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:40.414372 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:04:40.501420 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:40.503064 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:40.519999 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:40.522414 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:40.523103 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:40.524338 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:40.524402 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:40.528573 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:40.721033 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:04:40.723353 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:04:40.829044 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:40.832079 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:40.833941 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:40.836313 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:40.848253 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:40.852769 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:04:40.853260 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:40.857400 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:41.135856 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:41.156857 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:41.167450 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:41.168929 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:41.170637 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:41.175925 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:41.186601 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:41.187711 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:41.447376 7114 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 02:04:41.481909 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:41.484162 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:41.484935 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:41.491660 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:41.501997 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:41.508554 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:41.523298 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:41.634460 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 02:04:41.672904 6982 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:04:41.716177 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:41.805498 6982 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:04:41.810312 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:41.813108 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:41.815512 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:41.817296 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:41.820237 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 02:04:41.828666 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:41.829522 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:41.847210 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:42.138533 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:42.141124 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:42.141800 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:42.142208 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:42.143784 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:42.150125 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:42.156839 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:42.192008 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:42.209971 6849 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 02:04:42.249015 6982 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:04:42.454949 6982 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:04:42.482614 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:42.491576 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:42.495282 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:42.500656 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:42.506772 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:42.515314 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:42.521503 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:42.571714 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:42.803927 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:42.812412 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:42.814688 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:42.814743 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:42.831175 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:42.840834 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:42.842330 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 02:04:43.013836 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:04:43.053540 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:04:43.054234 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:04:43.060525 6718 consensus_peers.cc:462] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:43.120087 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 02:04:43.138425 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:43.139786 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:43.140127 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:43.143827 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:43.145802 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:43.146517 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:43.150568 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:43.326784 6849 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:04:43.462126 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:43.465143 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:43.468420 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:43.468920 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:43.483383 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:43.492110 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:43.503446 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:43.541700 6982 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): Couldn't send request to peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 02:04:43.647709 19157 raft_consensus.cc:998] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e: Attempting to remove follower 4e6e76b381af4cdc90bd3e9e6e8b5cf2 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 for more than 300 seconds (300.584s)
I20240502 02:04:43.648118 19157 consensus_queue.cc:237] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:04:43.649434 6916 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 02:04:43.649899 27559 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:04:43.651804 19157 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:04:43.653234 6916 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:04:43.655664 6642 catalog_manager.cc:5484] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
W20240502 02:04:43.661319 6627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:43.662874 6627 catalog_manager.cc:4573] TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0359dfbfd11746f1bf62a465f14634ec: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:43.665169 6783 consensus_queue.cc:237] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:04:43.666468 6916 raft_consensus.cc:1261] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 02:04:43.666736 6721 consensus_peers.cc:485] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:04:43.666934 19157 consensus_queue.cc:1035] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:04:43.668278 27559 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:04:43.668727 6916 raft_consensus.cc:2932] T 0359dfbfd11746f1bf62a465f14634ec P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:04:43.670303 6629 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0359dfbfd11746f1bf62a465f14634ec with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:04:43.670899 6643 catalog_manager.cc:5484] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 10 to 11, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:04:43.701328 27570 raft_consensus.cc:998] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8: Attempting to remove follower 4e6e76b381af4cdc90bd3e9e6e8b5cf2 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 for more than 300 seconds (300.266s)
I20240502 02:04:43.701782 27570 consensus_queue.cc:237] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 02:04:43.703652 7182 raft_consensus.cc:1261] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:04:43.703997 26963 consensus_queue.cc:1035] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:04:43.705588 27570 raft_consensus.cc:2932] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 02:04:43.706032 7182 raft_consensus.cc:2932] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 02:04:43.710749 6642 catalog_manager.cc:5484] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 5 to 6, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 02:04:43.717685 6916 consensus_queue.cc:237] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:43.719002 7182 raft_consensus.cc:1261] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 02:04:43.719120 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed7b67af7b1c4255a7c2377570b58e70. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:04:43.719429 27570 consensus_queue.cc:1035] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:04:43.720933 26963 raft_consensus.cc:2932] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.721347 7182 raft_consensus.cc:2932] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.722570 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ed7b67af7b1c4255a7c2377570b58e70 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:04:43.723888 6642 catalog_manager.cc:5484] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 6 to 7, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.727157 27570 raft_consensus.cc:998] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8: Attempting to remove follower 4e6e76b381af4cdc90bd3e9e6e8b5cf2 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 for more than 300 seconds (300.174s)
I20240502 02:04:43.727557 27570 consensus_queue.cc:237] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:04:43.728879 8904 raft_consensus.cc:1261] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:04:43.729305 27573 consensus_queue.cc:1035] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:04:43.731073 27570 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 02:04:43.732512 8904 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 02:04:43.733588 6643 catalog_manager.cc:5484] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 7 to 8, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 02:04:43.737020 6916 consensus_queue.cc:237] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:04:43.738466 8904 raft_consensus.cc:1261] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:04:43.738907 27570 consensus_queue.cc:1035] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:04:43.740476 27570 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:43.740934 8904 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:43.742447 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet dcc5daef41bc44c4855f2bcb44e1c6d5 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240502 02:04:43.743149 6851 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): Couldn't send request to peer 85e78945910a4e1abfa101d2edc3644e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcc5daef41bc44c4855f2bcb44e1c6d5. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:04:43.743516 6643 catalog_manager.cc:5484] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 070558c278e0457b9cc070ac0d662b5d reported cstate change: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
W20240502 02:04:43.754035 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 02:04:43.768404 27516 raft_consensus.cc:998] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5: Attempting to remove follower 4e6e76b381af4cdc90bd3e9e6e8b5cf2 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 for more than 300 seconds (300.387s)
I20240502 02:04:43.768816 27516 consensus_queue.cc:237] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:04:43.769290 26963 raft_consensus.cc:998] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8: Attempting to remove follower 4e6e76b381af4cdc90bd3e9e6e8b5cf2 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 for more than 300 seconds (300.295s)
I20240502 02:04:43.769644 26963 consensus_queue.cc:237] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:43.770720 8509 raft_consensus.cc:1261] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:04:43.770766 6783 raft_consensus.cc:1261] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:04:43.770802 8640 raft_consensus.cc:1261] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:04:43.771198 27570 consensus_queue.cc:1035] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:04:43.771203 27170 consensus_queue.cc:1035] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:04:43.771407 27516 consensus_queue.cc:1035] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:04:43.773587 27570 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.774879 8640 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.775003 27516 raft_consensus.cc:2932] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:04:43.775766 6783 raft_consensus.cc:2932] T b69d6e907b3b400f93721981732c006b P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:04:43.776611 8509 raft_consensus.cc:2932] T b69d6e907b3b400f93721981732c006b P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:04:43.776997 6642 catalog_manager.cc:5484] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 7 to 8, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.778844 6643 catalog_manager.cc:5484] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 6 to 7, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:04:43.781810 6916 consensus_queue.cc:237] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 02:04:43.784085 8640 raft_consensus.cc:1261] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:04:43.784503 27570 consensus_queue.cc:1035] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:04:43.786515 27573 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 02:04:43.787446 8640 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
W20240502 02:04:43.788432 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125): Couldn't send request to peer c6101231591a458384f4ea6d1f0e0ae5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:04:43.788908 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 818dbc9a8e054eeaa122e7919a311806 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:04:43.789983 6637 catalog_manager.cc:5484] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 8 to 9, VOTER c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 02:04:43.810133 27215 raft_consensus.cc:998] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c: Attempting to remove follower 4e6e76b381af4cdc90bd3e9e6e8b5cf2 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 for more than 300 seconds (300.532s)
I20240502 02:04:43.810612 27215 consensus_queue.cc:237] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:04:43.812554 6783 raft_consensus.cc:1261] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:04:43.813448 27215 consensus_queue.cc:1035] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:04:43.816990 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:43.817865 27215 raft_consensus.cc:2932] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:43.821065 6637 catalog_manager.cc:5484] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 5 to 6, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:43.821154 6783 raft_consensus.cc:2932] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:43.823143 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:43.826330 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:43.826857 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:43.831450 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:43.835126 7182 consensus_queue.cc:237] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:04:43.841280 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:43.842815 7114 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8529a93abd6e4d62bbc0bddcf2fda78a. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:04:43.843474 6783 raft_consensus.cc:1261] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:04:43.843880 27215 consensus_queue.cc:1035] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:04:43.860184 27215 raft_consensus.cc:2932] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 02:04:43.863996 6629 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8529a93abd6e4d62bbc0bddcf2fda78a with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:04:43.864257 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:43.864535 6783 raft_consensus.cc:2932] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 02:04:43.867368 6635 catalog_manager.cc:5484] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 6 to 7, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 02:04:43.905275 27628 raft_consensus.cc:998] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Attempting to remove follower 4e6e76b381af4cdc90bd3e9e6e8b5cf2 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 for more than 300 seconds (300.294s)
I20240502 02:04:43.905661 27628 consensus_queue.cc:237] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:43.907179 8640 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 02:04:43.907191 8509 raft_consensus.cc:1261] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 02:04:43.907591 27624 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 02:04:43.907765 27627 consensus_queue.cc:1035] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 02:04:43.909412 27628 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.909854 8640 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.909868 8509 raft_consensus.cc:2932] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.912642 6637 catalog_manager.cc:5484] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 12 to 13, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 3 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.920369 27170 raft_consensus.cc:998] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5: Attempting to remove follower 4e6e76b381af4cdc90bd3e9e6e8b5cf2 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 for more than 300 seconds (300.211s)
I20240502 02:04:43.920758 27170 consensus_queue.cc:237] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:04:43.921918 6783 raft_consensus.cc:1261] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:04:43.922300 27170 consensus_queue.cc:1035] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:04:43.923993 27516 raft_consensus.cc:2932] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:43.924430 6783 raft_consensus.cc:2932] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:43.926683 6636 catalog_manager.cc:5484] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 5 to 6, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:43.932807 7050 consensus_queue.cc:237] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:43.934197 6783 raft_consensus.cc:1261] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 02:04:43.934510 6985 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0e05f96dbc542eb999c16dcc0620666. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:04:43.934696 27516 consensus_queue.cc:1035] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:04:43.935992 27598 raft_consensus.cc:998] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5: Attempting to remove follower 4e6e76b381af4cdc90bd3e9e6e8b5cf2 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 for more than 300 seconds (300.466s)
I20240502 02:04:43.936363 27598 consensus_queue.cc:237] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:04:43.936691 27170 raft_consensus.cc:2932] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.937070 6783 raft_consensus.cc:2932] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.938714 6785 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 02:04:43.938714 8903 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 02:04:43.939159 27516 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:04:43.939283 27598 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:04:43.940218 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e0e05f96dbc542eb999c16dcc0620666 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:04:43.941259 6636 catalog_manager.cc:5484] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 6 to 7, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.941920 27637 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:43.942924 8904 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:43.943043 6783 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:43.943297 27634 raft_consensus.cc:998] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5: Attempting to remove follower 4e6e76b381af4cdc90bd3e9e6e8b5cf2 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 for more than 300 seconds (300.167s)
I20240502 02:04:43.943567 27634 consensus_queue.cc:237] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:43.945534 8640 raft_consensus.cc:1261] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 02:04:43.945720 6635 catalog_manager.cc:5484] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 2 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:43.946244 27634 consensus_queue.cc:1035] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:04:43.949223 27637 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.949620 8640 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.951269 6637 catalog_manager.cc:5484] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:43.954267 7050 consensus_queue.cc:237] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:04:43.955789 8640 raft_consensus.cc:1261] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 02:04:43.956178 27634 consensus_queue.cc:1035] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:04:43.957633 27637 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:04:43.958004 8640 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
W20240502 02:04:43.959251 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): Couldn't send request to peer e7bcff2fef614e7a9981554c9e6db3c8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:04:43.960083 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 14e3c78c65334b52aca79d67c4bf45a2 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:04:43.960772 6637 catalog_manager.cc:5484] T 14e3c78c65334b52aca79d67c4bf45a2 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 10 to 11, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:04:44.114303 27636 raft_consensus.cc:998] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5: Attempting to remove follower 4e6e76b381af4cdc90bd3e9e6e8b5cf2 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 for more than 300 seconds (300.322s)
I20240502 02:04:44.114722 27636 consensus_queue.cc:237] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:44.115952 8640 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 02:04:44.116343 27634 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:04:44.118062 27634 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:44.120188 8640 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:44.122192 6636 catalog_manager.cc:5484] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 9 to 10, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 3 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:44.125638 7050 consensus_queue.cc:237] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
W20240502 02:04:44.127076 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): Couldn't send request to peer e7bcff2fef614e7a9981554c9e6db3c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91db5c1b3d2f4d22b5882a61ea0395c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:04:44.127153 8640 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 02:04:44.127710 27636 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:04:44.129354 27634 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:04:44.129703 8640 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P 4f53d80e39be410d92d4766692aff1c6 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:04:44.131356 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 91db5c1b3d2f4d22b5882a61ea0395c7 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:04:44.132135 6636 catalog_manager.cc:5484] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 10 to 11, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New cstate: current_term: 3 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:04:44.133587 19157 raft_consensus.cc:998] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e: Attempting to remove follower 4e6e76b381af4cdc90bd3e9e6e8b5cf2 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 for more than 300 seconds (300.179s)
I20240502 02:04:44.133898 19157 consensus_queue.cc:237] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:44.135037 8640 raft_consensus.cc:1261] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:04:44.135392 27559 consensus_queue.cc:1035] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:04:44.137452 27559 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:44.139190 8640 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:44.141328 6636 catalog_manager.cc:5484] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 7 to 8, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:44.146307 6783 consensus_queue.cc:237] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:04:44.148311 8640 raft_consensus.cc:1261] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer 85e78945910a4e1abfa101d2edc3644e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:04:44.148741 27559 consensus_queue.cc:1035] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:04:44.150489 19157 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 02:04:44.152891 6629 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a4fb9af11abc49d496b5c3d8048c22c8 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:04:44.154022 8640 raft_consensus.cc:2932] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 02:04:44.157457 6640 catalog_manager.cc:5484] T a4fb9af11abc49d496b5c3d8048c22c8 P 4f53d80e39be410d92d4766692aff1c6 reported cstate change: config changed from index 8 to 9, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 1 leader_uuid: "85e78945910a4e1abfa101d2edc3644e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
W20240502 02:04:44.160432 6718 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4fb9af11abc49d496b5c3d8048c22c8. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:04:44.170598 27678 ts_tablet_manager.cc:902] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 02:04:44.171542 27678 tablet_copy_client.cc:308] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 02:04:44.173295 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet 818dbc9a8e054eeaa122e7919a311806 from peer c6101231591a458384f4ea6d1f0e0ae5 ({username='slave'} at 127.27.111.195:40287)
I20240502 02:04:44.173446 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet 818dbc9a8e054eeaa122e7919a311806 from peer c6101231591a458384f4ea6d1f0e0ae5 at {username='slave'} at 127.27.111.195:40287: session id = c6101231591a458384f4ea6d1f0e0ae5-818dbc9a8e054eeaa122e7919a311806
I20240502 02:04:44.175024 6937 tablet_copy_source_session.cc:205] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:04:44.176074 27678 ts_tablet_manager.cc:1875] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:04:44.178925 27678 ts_tablet_manager.cc:1888] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 02:04:44.179114 27678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818dbc9a8e054eeaa122e7919a311806. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:04:44.180264 27678 tablet_copy_client.cc:783] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: tablet copy: Starting download of 5 data blocks...
I20240502 02:04:44.182822 27694 ts_tablet_manager.cc:902] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261)
I20240502 02:04:44.183360 8884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:44.184844 27694 tablet_copy_client.cc:308] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:42261
I20240502 02:04:44.188845 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:44.189620 6805 tablet_copy_service.cc:140] P 85e78945910a4e1abfa101d2edc3644e: Received BeginTabletCopySession request for tablet 0359dfbfd11746f1bf62a465f14634ec from peer 954cd49326bf405abd81182d75dc783c ({username='slave'} at 127.27.111.197:36883)
I20240502 02:04:44.189713 6805 tablet_copy_service.cc:161] P 85e78945910a4e1abfa101d2edc3644e: Beginning new tablet copy session on tablet 0359dfbfd11746f1bf62a465f14634ec from peer 954cd49326bf405abd81182d75dc783c at {username='slave'} at 127.27.111.197:36883: session id = 954cd49326bf405abd81182d75dc783c-0359dfbfd11746f1bf62a465f14634ec
I20240502 02:04:44.190774 27678 tablet_copy_client.cc:647] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: tablet copy: Starting download of 1 WAL segments...
I20240502 02:04:44.191195 6805 tablet_copy_source_session.cc:205] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:04:44.195181 27678 tablet_copy_client.cc:523] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:04:44.195228 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:44.197000 27678 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap starting.
I20240502 02:04:44.192417 27694 ts_tablet_manager.cc:1875] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:04:44.203837 27694 ts_tablet_manager.cc:1888] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240502 02:04:44.204094 27694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0359dfbfd11746f1bf62a465f14634ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:04:44.205396 27694 tablet_copy_client.cc:783] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 5 data blocks...
I20240502 02:04:44.206054 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:44.209481 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:44.219631 8621 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:44.219897 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:44.229722 27678 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:04:44.231534 27678 tablet_bootstrap.cc:492] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: Bootstrap complete.
I20240502 02:04:44.231760 27678 ts_tablet_manager.cc:1367] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.000s
I20240502 02:04:44.232208 27678 raft_consensus.cc:348] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 02:04:44.232452 27678 raft_consensus.cc:729] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6101231591a458384f4ea6d1f0e0ae5, State: Initialized, Role: FOLLOWER
I20240502 02:04:44.232617 27678 consensus_queue.cc:260] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 02:04:44.232929 27678 ts_tablet_manager.cc:1398] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:04:44.233436 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session c6101231591a458384f4ea6d1f0e0ae5-818dbc9a8e054eeaa122e7919a311806 received from {username='slave'} at 127.27.111.195:40287
I20240502 02:04:44.233525 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session c6101231591a458384f4ea6d1f0e0ae5-818dbc9a8e054eeaa122e7919a311806 on tablet 818dbc9a8e054eeaa122e7919a311806 with peer c6101231591a458384f4ea6d1f0e0ae5
I20240502 02:04:44.237794 27694 tablet_copy_client.cc:647] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: tablet copy: Starting download of 1 WAL segments...
I20240502 02:04:44.240458 27694 tablet_copy_client.cc:523] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:04:44.242172 27694 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Bootstrap starting.
I20240502 02:04:44.252856 27694 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:04:44.253144 27694 tablet_bootstrap.cc:492] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Bootstrap complete.
I20240502 02:04:44.253341 27694 ts_tablet_manager.cc:1367] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 02:04:44.253947 27694 raft_consensus.cc:348] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:04:44.254099 27694 raft_consensus.cc:729] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954cd49326bf405abd81182d75dc783c, State: Initialized, Role: FOLLOWER
I20240502 02:04:44.254289 27694 consensus_queue.cc:260] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:04:44.254706 27694 ts_tablet_manager.cc:1398] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 02:04:44.256187 6805 tablet_copy_service.cc:342] P 85e78945910a4e1abfa101d2edc3644e: Request end of tablet copy session 954cd49326bf405abd81182d75dc783c-0359dfbfd11746f1bf62a465f14634ec received from {username='slave'} at 127.27.111.197:36883
I20240502 02:04:44.256278 6805 tablet_copy_service.cc:433] P 85e78945910a4e1abfa101d2edc3644e: ending tablet copy session 954cd49326bf405abd81182d75dc783c-0359dfbfd11746f1bf62a465f14634ec on tablet 0359dfbfd11746f1bf62a465f14634ec with peer 954cd49326bf405abd81182d75dc783c
I20240502 02:04:44.270270 27636 raft_consensus.cc:998] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5: Attempting to remove follower 4e6e76b381af4cdc90bd3e9e6e8b5cf2 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4e6e76b381af4cdc90bd3e9e6e8b5cf2 for more than 300 seconds (300.512s)
I20240502 02:04:44.270705 27636 consensus_queue.cc:237] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:44.272094 8509 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:04:44.272135 8640 raft_consensus.cc:1261] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:04:44.272715 27637 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:04:44.272908 27598 consensus_queue.cc:1035] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:04:44.275053 27634 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:44.275197 7050 consensus_queue.cc:237] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:04:44.275888 8640 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:44.276245 6783 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 02:04:44.276585 8509 raft_consensus.cc:2932] T 48eeb721a60c4be888041bd31cfeab7a P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:44.277235 27598 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:04:44.277906 6637 catalog_manager.cc:5484] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 8 to 9, VOTER 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } }
I20240502 02:04:44.280285 27598 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:44.280634 6783 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:44.283511 6642 catalog_manager.cc:5484] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 10 to 11, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New cstate: current_term: 2 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:44.286563 7050 consensus_queue.cc:237] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:04:44.287793 8904 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 02:04:44.287793 6783 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 02:04:44.288185 27637 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:04:44.288707 27637 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 02:04:44.289367 8904 raft_consensus.cc:858] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d [term 2 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }:
New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
W20240502 02:04:44.289443 8904 raft_consensus.cc:1565] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d [term 2 NON_PARTICIPANT]: Could not prepare op 'term: 2 index: 12' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }:
New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:04:44.289733 8904 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d [term 2 NON_PARTICIPANT]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:04:44.289942 8884 tablet_service.cc:1514] Processing DeleteTablet for tablet a0548274c47148d9b611fd3c978a0c34 with delete_type TABLET_DATA_TOMBSTONED (TS 070558c278e0457b9cc070ac0d662b5d not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:42392
W20240502 02:04:44.291175 6627 catalog_manager.cc:4804] TS 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): delete failed for tablet a0548274c47148d9b611fd3c978a0c34 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 10 but the committed config has opid_index of 11
I20240502 02:04:44.292270 8904 raft_consensus.cc:1467] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d [term 2 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240502 02:04:44.292510 27634 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 02:04:44.292822 6783 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 02:04:44.293612 8904 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 070558c278e0457b9cc070ac0d662b5d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 02:04:44.293843 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a0548274c47148d9b611fd3c978a0c34 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:04:44.295487 6642 catalog_manager.cc:5484] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 11 to 12, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) added. New cstate: current_term: 2 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } }
I20240502 02:04:44.301333 27721 ts_tablet_manager.cc:902] T ed7b67af7b1c4255a7c2377570b58e70 P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 02:04:44.301788 27721 tablet_copy_client.cc:308] T ed7b67af7b1c4255a7c2377570b58e70 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 02:04:44.304140 27727 ts_tablet_manager.cc:902] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 02:04:44.304669 27727 tablet_copy_client.cc:308] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 02:04:44.305111 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet dcc5daef41bc44c4855f2bcb44e1c6d5 from peer 85e78945910a4e1abfa101d2edc3644e ({username='slave'} at 127.27.111.193:40059)
I20240502 02:04:44.305218 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet dcc5daef41bc44c4855f2bcb44e1c6d5 from peer 85e78945910a4e1abfa101d2edc3644e at {username='slave'} at 127.27.111.193:40059: session id = 85e78945910a4e1abfa101d2edc3644e-dcc5daef41bc44c4855f2bcb44e1c6d5
I20240502 02:04:44.305917 6936 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet ed7b67af7b1c4255a7c2377570b58e70 from peer 4f53d80e39be410d92d4766692aff1c6 ({username='slave'} at 127.27.111.198:57121)
I20240502 02:04:44.306030 6936 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet ed7b67af7b1c4255a7c2377570b58e70 from peer 4f53d80e39be410d92d4766692aff1c6 at {username='slave'} at 127.27.111.198:57121: session id = 4f53d80e39be410d92d4766692aff1c6-ed7b67af7b1c4255a7c2377570b58e70
I20240502 02:04:44.306304 6937 tablet_copy_source_session.cc:205] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:04:44.307006 6936 tablet_copy_source_session.cc:205] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:04:44.307175 27727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc5daef41bc44c4855f2bcb44e1c6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:04:44.308034 27721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed7b67af7b1c4255a7c2377570b58e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:04:44.310335 27721 tablet_copy_client.cc:783] T ed7b67af7b1c4255a7c2377570b58e70 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 5 data blocks...
I20240502 02:04:44.310840 27727 tablet_copy_client.cc:783] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 5 data blocks...
I20240502 02:04:44.314738 27721 tablet_copy_client.cc:647] T ed7b67af7b1c4255a7c2377570b58e70 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Starting download of 1 WAL segments...
I20240502 02:04:44.315649 27727 tablet_copy_client.cc:647] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 1 WAL segments...
I20240502 02:04:44.315997 27721 tablet_copy_client.cc:523] T ed7b67af7b1c4255a7c2377570b58e70 P 4f53d80e39be410d92d4766692aff1c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:04:44.317409 27727 tablet_copy_client.cc:523] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:04:44.317538 27721 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap starting.
I20240502 02:04:44.318938 27727 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 02:04:44.326851 27721 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:04:44.327289 27721 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P 4f53d80e39be410d92d4766692aff1c6: Bootstrap complete.
I20240502 02:04:44.327502 27721 ts_tablet_manager.cc:1367] T ed7b67af7b1c4255a7c2377570b58e70 P 4f53d80e39be410d92d4766692aff1c6: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.004s
I20240502 02:04:44.328018 27721 raft_consensus.cc:348] T ed7b67af7b1c4255a7c2377570b58e70 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:44.328155 27721 raft_consensus.cc:729] T ed7b67af7b1c4255a7c2377570b58e70 P 4f53d80e39be410d92d4766692aff1c6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53d80e39be410d92d4766692aff1c6, State: Initialized, Role: FOLLOWER
I20240502 02:04:44.328315 27721 consensus_queue.cc:260] T ed7b67af7b1c4255a7c2377570b58e70 P 4f53d80e39be410d92d4766692aff1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:44.328482 27727 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:04:44.328644 27721 ts_tablet_manager.cc:1398] T ed7b67af7b1c4255a7c2377570b58e70 P 4f53d80e39be410d92d4766692aff1c6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:04:44.328786 27727 tablet_bootstrap.cc:492] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap complete.
I20240502 02:04:44.329151 27727 ts_tablet_manager.cc:1367] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20240502 02:04:44.329433 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 4f53d80e39be410d92d4766692aff1c6-ed7b67af7b1c4255a7c2377570b58e70 received from {username='slave'} at 127.27.111.198:57121
I20240502 02:04:44.329527 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 4f53d80e39be410d92d4766692aff1c6-ed7b67af7b1c4255a7c2377570b58e70 on tablet ed7b67af7b1c4255a7c2377570b58e70 with peer 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:44.329566 27727 raft_consensus.cc:348] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:04:44.329696 27727 raft_consensus.cc:729] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 02:04:44.329856 27727 consensus_queue.cc:260] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:04:44.330130 27727 ts_tablet_manager.cc:1398] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:04:44.331095 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 85e78945910a4e1abfa101d2edc3644e-dcc5daef41bc44c4855f2bcb44e1c6d5 received from {username='slave'} at 127.27.111.193:40059
I20240502 02:04:44.331180 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 85e78945910a4e1abfa101d2edc3644e-dcc5daef41bc44c4855f2bcb44e1c6d5 on tablet dcc5daef41bc44c4855f2bcb44e1c6d5 with peer 85e78945910a4e1abfa101d2edc3644e
I20240502 02:04:44.366971 27721 ts_tablet_manager.cc:902] T e0e05f96dbc542eb999c16dcc0620666 P 4f53d80e39be410d92d4766692aff1c6: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
F20240502 02:04:44.367163 27721 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.queue_time_us, tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.queue_time_us) (tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.run_cpu_time_us, tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.run_cpu_time_us) (tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.run_wall_time_us, tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.run_wall_time_us) (tablet-download-pool-14e3c78c65334b52aca79d67c4bf45a2.queue_time_us, tablet-download-pool-14e3c78c65334b52aca79d67c4bf45a2.queue_time_us) (tablet-download-pool-14e3c78c65334b52aca79d67c4bf45a2.run_cpu_time_us, tablet-download-pool-14e3c78c65334b52aca79d67c4bf45a2.run_cpu_time_us) (tablet-download-pool-14e3c78c65334b52aca79d67c4bf45a2.run_wall_time_us, tablet-download-pool-14e3c78c65334b52aca79d67c4bf45a2.run_wall_time_us) (tablet-download-pool-17235e019e644b258266636fb6be6648.queue_time_us, tablet-download-pool-17235e019e644b258266636fb6be6648.queue_time_us) (tablet-download-pool-17235e019e644b258266636fb6be6648.run_cpu_time_us, tablet-download-pool-17235e019e644b258266636fb6be6648.run_cpu_time_us) (tablet-download-pool-17235e019e644b258266636fb6be6648.run_wall_time_us, tablet-download-pool-17235e019e644b258266636fb6be6648.run_wall_time_us) (tablet-download-pool-39d9e48f67f34ebe850fc388ecd94ca3.queue_time_us, tablet-download-pool-39d9e48f67f34ebe850fc388ecd94ca3.queue_time_us) (tablet-download-pool-39d9e48f67f34ebe850fc388ecd94ca3.run_cpu_time_us, tablet-download-pool-39d9e48f67f34ebe850fc388ecd94ca3.run_cpu_time_us) (tablet-download-pool-39d9e48f67f34ebe850fc388ecd94ca3.run_wall_time_us, tablet-download-pool-39d9e48f67f34ebe850fc388ecd94ca3.run_wall_time_us) (tablet-download-pool-47a276e9780e448a9b3c6eab67925514.queue_time_us, tablet-download-pool-47a276e9780e448a9b3c6eab67925514.queue_time_us) (tablet-download-pool-47a276e9780e448a9b3c6eab67925514.run_cpu_time_us, tablet-download-pool-47a276e9780e448a9b3c6eab67925514.run_cpu_time_us) (tablet-download-pool-47a276e9780e448a9b3c6eab67925514.run_wall_time_us, tablet-download-pool-47a276e9780e448a9b3c6eab67925514.run_wall_time_us) (tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.queue_time_us, tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.queue_time_us) (tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.run_cpu_time_us, tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.run_cpu_time_us) (tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.run_wall_time_us, tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.run_wall_time_us) (tablet-download-pool-818dbc9a8e054eeaa122e7919a311806.queue_time_us, tablet-download-pool-818dbc9a8e054eeaa122e7919a311806.queue_time_us) (tablet-download-pool-818dbc9a8e054eeaa122e7919a311806.run_cpu_time_us, tablet-download-pool-818dbc9a8e054eeaa122e7919a311806.run_cpu_time_us) (tablet-download-pool-818dbc9a8e054eeaa122e7919a311806.run_wall_time_us, tablet-download-pool-818dbc9a8e054eeaa122e7919a311806.run_wall_time_us) (tablet-download-pool-91db5c1b3d2f4d22b5882a61ea0395c7.queue_time_us, tablet-download-pool-91db5c1b3d2f4d22b5882a61ea0395c7.queue_time_us) (tablet-download-pool-91db5c1b3d2f4d22b5882a61ea0395c7.run_cpu_time_us, tablet-download-pool-91db5c1b3d2f4d22b5882a61ea0395c7.run_cpu_time_us) (tablet-download-pool-91db5c1b3d2f4d22b5882a61ea0395c7.run_wall_time_us, tablet-download-pool-91db5c1b3d2f4d22b5882a61ea0395c7.run_wall_time_us) (tablet-download-pool-9f6c07e3923d47f29ef32561d7f438e1.queue_time_us, tablet-download-pool-9f6c07e3923d47f29ef32561d7f438e1.queue_time_us) (tablet-download-pool-9f6c07e3923d47f29ef32561d7f438e1.run_cpu_time_us, tablet-download-pool-9f6c07e3923d47f29ef32561d7f438e1.run_cpu_time_us) (tablet-download-pool-9f6c07e3923d47f29ef32561d7f438e1.run_wall_time_us, tablet-download-pool-9f6c07e3923d47f29ef32561d7f438e1.run_wall_time_us) (tablet-download-pool-a4fb9af11abc49d496b5c3d8048c22c8.queue_time_us, tablet-download-pool-a4fb9af11abc49d496b5c3d8048c22c8.queue_time_us) (tablet-download-pool-a4fb9af11abc49d496b5c3d8048c22c8.run_cpu_time_us, tablet-download-pool-a4fb9af11abc49d496b5c3d8048c22c8.run_cpu_time_us) (tablet-download-pool-a4fb9af11abc49d496b5c3d8048c22c8.run_wall_time_us, tablet-download-pool-a4fb9af11abc49d496b5c3d8048c22c8.run_wall_time_us) (tablet-download-pool-a66b932d884e4df8a78841103d88ded0.queue_time_us, tablet-download-pool-a66b932d884e4df8a78841103d88ded0.queue_time_us) (tablet-download-pool-a66b932d884e4df8a78841103d88ded0.run_cpu_time_us, tablet-download-pool-a66b932d884e4df8a78841103d88ded0.run_cpu_time_us) (tablet-download-pool-a66b932d884e4df8a78841103d88ded0.run_wall_time_us, tablet-download-pool-a66b932d884e4df8a78841103d88ded0.run_wall_time_us) (tablet-download-pool-c8c237c50773494abf8813494be0f184.queue_time_us, tablet-download-pool-c8c237c50773494abf8813494be0f184.queue_time_us) (tablet-download-pool-c8c237c50773494abf8813494be0f184.run_cpu_time_us, tablet-download-pool-c8c237c50773494abf8813494be0f184.run_cpu_time_us) (tablet-download-pool-c8c237c50773494abf8813494be0f184.run_wall_time_us, tablet-download-pool-c8c237c50773494abf8813494be0f184.run_wall_time_us) (tablet-download-pool-e0e05f96dbc542eb999c16dcc0620666.queue_time_us, tablet-download-pool-e0e05f96dbc542eb999c16dcc0620666.queue_time_us) (tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.queue_time_us, tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.queue_time_us) (tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.run_cpu_time_us, tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.run_cpu_time_us) (tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.run_wall_time_us, tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.run_wall_time_us) (tablet-download-pool-ed7b67af7b1c4255a7c2377570b58e70.queue_time_us, tablet-download-pool-ed7b67af7b1c4255a7c2377570b58e70.queue_time_us) (tablet-download-pool-ed7b67af7b1c4255a7c2377570b58e70.run_cpu_time_us, tablet-download-pool-ed7b67af7b1c4255a7c2377570b58e70.run_cpu_time_us) (tablet-download-pool-ed7b67af7b1c4255a7c2377570b58e70.run_wall_time_us, tablet-download-pool-ed7b67af7b1c4255a7c2377570b58e70.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f3d52047dcd google::LogMessage::Fail() at ??:0
@ 0x7f3d5204bb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f3d520477cc google::LogMessage::Flush() at ??:0
I20240502 02:04:44.374960 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling UndoDeltaBlockGCOp(dcc5daef41bc44c4855f2bcb44e1c6d5): 4921 bytes on disk
I20240502 02:04:44.375526 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: UndoDeltaBlockGCOp(dcc5daef41bc44c4855f2bcb44e1c6d5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240502 02:04:44.375859 27736 ts_tablet_manager.cc:902] T 8529a93abd6e4d62bbc0bddcf2fda78a P 070558c278e0457b9cc070ac0d662b5d: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
F20240502 02:04:44.376041 27736 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.queue_time_us, tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.queue_time_us) (tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.run_cpu_time_us, tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.run_cpu_time_us) (tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.run_wall_time_us, tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.run_wall_time_us) (tablet-download-pool-0359dfbfd11746f1bf62a465f14634ec.queue_time_us, tablet-download-pool-0359dfbfd11746f1bf62a465f14634ec.queue_time_us) (tablet-download-pool-0359dfbfd11746f1bf62a465f14634ec.run_cpu_time_us, tablet-download-pool-0359dfbfd11746f1bf62a465f14634ec.run_cpu_time_us) (tablet-download-pool-0359dfbfd11746f1bf62a465f14634ec.run_wall_time_us, tablet-download-pool-0359dfbfd11746f1bf62a465f14634ec.run_wall_time_us) (tablet-download-pool-1c54b912f802461c8cac092ac3222c87.queue_time_us, tablet-download-pool-1c54b912f802461c8cac092ac3222c87.queue_time_us) (tablet-download-pool-1c54b912f802461c8cac092ac3222c87.run_cpu_time_us, tablet-download-pool-1c54b912f802461c8cac092ac3222c87.run_cpu_time_us) (tablet-download-pool-1c54b912f802461c8cac092ac3222c87.run_wall_time_us, tablet-download-pool-1c54b912f802461c8cac092ac3222c87.run_wall_time_us) (tablet-download-pool-347b161e9ec84640a0668836ce9e29eb.queue_time_us, tablet-download-pool-347b161e9ec84640a0668836ce9e29eb.queue_time_us) (tablet-download-pool-347b161e9ec84640a0668836ce9e29eb.run_cpu_time_us, tablet-download-pool-347b161e9ec84640a0668836ce9e29eb.run_cpu_time_us) (tablet-download-pool-347b161e9ec84640a0668836ce9e29eb.run_wall_time_us, tablet-download-pool-347b161e9ec84640a0668836ce9e29eb.run_wall_time_us) (tablet-download-pool-39d9e48f67f34ebe850fc388ecd94ca3.queue_time_us, tablet-download-pool-39d9e48f67f34ebe850fc388ecd94ca3.queue_time_us) (tablet-download-pool-39d9e48f67f34ebe850fc388ecd94ca3.run_cpu_time_us, tablet-download-pool-39d9e48f67f34ebe850fc388ecd94ca3.run_cpu_time_us) (tablet-download-pool-39d9e48f67f34ebe850fc388ecd94ca3.run_wall_time_us, tablet-download-pool-39d9e48f67f34ebe850fc388ecd94ca3.run_wall_time_us) (tablet-download-pool-4154be3f1f3c428580463c419d69892e.queue_time_us, tablet-download-pool-4154be3f1f3c428580463c419d69892e.queue_time_us) (tablet-download-pool-4154be3f1f3c428580463c419d69892e.run_cpu_time_us, tablet-download-pool-4154be3f1f3c428580463c419d69892e.run_cpu_time_us) (tablet-download-pool-4154be3f1f3c428580463c419d69892e.run_wall_time_us, tablet-download-pool-4154be3f1f3c428580463c419d69892e.run_wall_time_us) (tablet-download-pool-44c55b199a24432e902cce9121fc0780.queue_time_us, tablet-download-pool-44c55b199a24432e902cce9121fc0780.queue_time_us) (tablet-download-pool-44c55b199a24432e902cce9121fc0780.run_cpu_time_us, tablet-download-pool-44c55b199a24432e902cce9121fc0780.run_cpu_time_us) (tablet-download-pool-44c55b199a24432e902cce9121fc0780.run_wall_time_us, tablet-download-pool-44c55b199a24432e902cce9121fc0780.run_wall_time_us) (tablet-download-pool-7345f51bd6f84b3880838c2460bafb75.queue_time_us, tablet-download-pool-7345f51bd6f84b3880838c2460bafb75.queue_time_us) (tablet-download-pool-7345f51bd6f84b3880838c2460bafb75.run_cpu_time_us, tablet-download-pool-7345f51bd6f84b3880838c2460bafb75.run_cpu_time_us) (tablet-download-pool-7345f51bd6f84b3880838c2460bafb75.run_wall_time_us, tablet-download-pool-7345f51bd6f84b3880838c2460bafb75.run_wall_time_us) (tablet-download-pool-7fa358140eb742b587a4d6a203738b4f.queue_time_us, tablet-download-pool-7fa358140eb742b587a4d6a203738b4f.queue_time_us) (tablet-download-pool-7fa358140eb742b587a4d6a203738b4f.run_cpu_time_us, tablet-download-pool-7fa358140eb742b587a4d6a203738b4f.run_cpu_time_us) (tablet-download-pool-7fa358140eb742b587a4d6a203738b4f.run_wall_time_us, tablet-download-pool-7fa358140eb742b587a4d6a203738b4f.run_wall_time_us) (tablet-download-pool-8529a93abd6e4d62bbc0bddcf2fda78a.queue_time_us, tablet-download-pool-8529a93abd6e4d62bbc0bddcf2fda78a.queue_time_us) (tablet-download-pool-9ecbf1c8ee8c46b195d2374a5db81ac0.queue_time_us, tablet-download-pool-9ecbf1c8ee8c46b195d2374a5db81ac0.queue_time_us) (tablet-download-pool-9ecbf1c8ee8c46b195d2374a5db81ac0.run_cpu_time_us, tablet-download-pool-9ecbf1c8ee8c46b195d2374a5db81ac0.run_cpu_time_us) (tablet-download-pool-9ecbf1c8ee8c46b195d2374a5db81ac0.run_wall_time_us, tablet-download-pool-9ecbf1c8ee8c46b195d2374a5db81ac0.run_wall_time_us) (tablet-download-pool-9f6c07e3923d47f29ef32561d7f438e1.queue_time_us, tablet-download-pool-9f6c07e3923d47f29ef32561d7f438e1.queue_time_us) (tablet-download-pool-9f6c07e3923d47f29ef32561d7f438e1.run_cpu_time_us, tablet-download-pool-9f6c07e3923d47f29ef32561d7f438e1.run_cpu_time_us) (tablet-download-pool-9f6c07e3923d47f29ef32561d7f438e1.run_wall_time_us, tablet-download-pool-9f6c07e3923d47f29ef32561d7f438e1.run_wall_time_us) (tablet-download-pool-a0548274c47148d9b611fd3c978a0c34.queue_time_us, tablet-download-pool-a0548274c47148d9b611fd3c978a0c34.queue_time_us) (tablet-download-pool-a0548274c47148d9b611fd3c978a0c34.run_cpu_time_us, tablet-download-pool-a0548274c47148d9b611fd3c978a0c34.run_cpu_time_us) (tablet-download-pool-a0548274c47148d9b611fd3c978a0c34.run_wall_time_us, tablet-download-pool-a0548274c47148d9b611fd3c978a0c34.run_wall_time_us) (tablet-download-pool-c8c237c50773494abf8813494be0f184.queue_time_us, tablet-download-pool-c8c237c50773494abf8813494be0f184.queue_time_us) (tablet-download-pool-c8c237c50773494abf8813494be0f184.run_cpu_time_us, tablet-download-pool-c8c237c50773494abf8813494be0f184.run_cpu_time_us) (tablet-download-pool-c8c237c50773494abf8813494be0f184.run_wall_time_us, tablet-download-pool-c8c237c50773494abf8813494be0f184.run_wall_time_us) (tablet-download-pool-dcc5daef41bc44c4855f2bcb44e1c6d5.queue_time_us, tablet-download-pool-dcc5daef41bc44c4855f2bcb44e1c6d5.queue_time_us) (tablet-download-pool-dcc5daef41bc44c4855f2bcb44e1c6d5.run_cpu_time_us, tablet-download-pool-dcc5daef41bc44c4855f2bcb44e1c6d5.run_cpu_time_us) (tablet-download-pool-dcc5daef41bc44c4855f2bcb44e1c6d5.run_wall_time_us, tablet-download-pool-dcc5daef41bc44c4855f2bcb44e1c6d5.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f3d52048f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 02:04:44.377336 8556 maintenance_manager.cc:419] P 954cd49326bf405abd81182d75dc783c: Scheduling UndoDeltaBlockGCOp(0359dfbfd11746f1bf62a465f14634ec): 4834 bytes on disk
I20240502 02:04:44.377856 8446 maintenance_manager.cc:643] P 954cd49326bf405abd81182d75dc783c: UndoDeltaBlockGCOp(0359dfbfd11746f1bf62a465f14634ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
@ 0x7f9308bffdcd google::LogMessage::Fail() at ??:0
@ 0x7f3d534b1044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f9308c03b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f3d53495f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f9308bff7cc google::LogMessage::Flush() at ??:0
@ 0x7f9308c00f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3d53493efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 02:04:44.388464 8689 maintenance_manager.cc:419] P 4f53d80e39be410d92d4766692aff1c6: Scheduling UndoDeltaBlockGCOp(ed7b67af7b1c4255a7c2377570b58e70): 4581 bytes on disk
@ 0x7f930a069044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f3d5604677a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 02:04:44.390915 8578 maintenance_manager.cc:643] P 4f53d80e39be410d92d4766692aff1c6: UndoDeltaBlockGCOp(ed7b67af7b1c4255a7c2377570b58e70) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
@ 0x7f930a04df98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f3d5604fa45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f930a04befc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f3d560d076f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f930cbfe77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f3d560e44f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f930cc07a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f3d560cf226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f930cc8876f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f3d560dec0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56484457075c std::function<>::operator()() at ??:0
@ 0x7f930cc9c4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f3d53498e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f930cc87226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240502 02:04:44.414453 7097 maintenance_manager.cc:419] P c6101231591a458384f4ea6d1f0e0ae5: Scheduling UndoDeltaBlockGCOp(818dbc9a8e054eeaa122e7919a311806): 4834 bytes on disk
I20240502 02:04:44.415196 6986 maintenance_manager.cc:643] P c6101231591a458384f4ea6d1f0e0ae5: UndoDeltaBlockGCOp(818dbc9a8e054eeaa122e7919a311806) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
@ 0x7f3d534996ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f930cc96c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55efed33175c std::function<>::operator()() at ??:0
@ 0x7f3d5349b03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56484457075c std::function<>::operator()() at ??:0
@ 0x7f930a050e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3d53488807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f930a0516ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3d53cc66db start_thread at ??:0
@ 0x7f930a05303b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55efed33175c std::function<>::operator()() at ??:0
@ 0x7f3d511fe71f clone at ??:0
@ 0x7f930a040807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f930a87e6db start_thread at ??:0
@ 0x7f9307db671f clone at ??:0
I20240502 02:04:44.452252 27750 ts_tablet_manager.cc:902] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 02:04:44.452812 27750 tablet_copy_client.cc:308] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 02:04:44.453459 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 14e3c78c65334b52aca79d67c4bf45a2 from peer e7bcff2fef614e7a9981554c9e6db3c8 ({username='slave'} at 127.27.111.194:33611)
I20240502 02:04:44.453572 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 14e3c78c65334b52aca79d67c4bf45a2 from peer e7bcff2fef614e7a9981554c9e6db3c8 at {username='slave'} at 127.27.111.194:33611: session id = e7bcff2fef614e7a9981554c9e6db3c8-14e3c78c65334b52aca79d67c4bf45a2
I20240502 02:04:44.455168 7070 tablet_copy_source_session.cc:205] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:04:44.456192 27750 ts_tablet_manager.cc:1875] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:04:44.458957 27750 ts_tablet_manager.cc:1888] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 02:04:44.459169 27750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e3c78c65334b52aca79d67c4bf45a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:04:44.460487 27750 tablet_copy_client.cc:783] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Starting download of 5 data blocks...
I20240502 02:04:44.470086 27750 tablet_copy_client.cc:647] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Starting download of 1 WAL segments...
I20240502 02:04:44.472435 27750 tablet_copy_client.cc:523] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:04:44.474588 27750 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 02:04:44.500471 27750 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:04:44.500835 27750 tablet_bootstrap.cc:492] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap complete.
I20240502 02:04:44.501497 27750 ts_tablet_manager.cc:1367] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.000s
I20240502 02:04:44.502048 27750 raft_consensus.cc:348] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:04:44.502175 27750 raft_consensus.cc:729] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 02:04:44.502331 27750 consensus_queue.cc:260] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:04:44.503355 27750 ts_tablet_manager.cc:1398] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 02:04:44.503934 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session e7bcff2fef614e7a9981554c9e6db3c8-14e3c78c65334b52aca79d67c4bf45a2 received from {username='slave'} at 127.27.111.194:33611
I20240502 02:04:44.504020 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session e7bcff2fef614e7a9981554c9e6db3c8-14e3c78c65334b52aca79d67c4bf45a2 on tablet 14e3c78c65334b52aca79d67c4bf45a2 with peer e7bcff2fef614e7a9981554c9e6db3c8
I20240502 02:04:44.560184 8509 raft_consensus.cc:1203] T 0359dfbfd11746f1bf62a465f14634ec P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 02:04:44.567413 27750 ts_tablet_manager.cc:902] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 02:04:44.567968 27750 tablet_copy_client.cc:308] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 02:04:44.568645 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 91db5c1b3d2f4d22b5882a61ea0395c7 from peer e7bcff2fef614e7a9981554c9e6db3c8 ({username='slave'} at 127.27.111.194:33611)
I20240502 02:04:44.568771 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 91db5c1b3d2f4d22b5882a61ea0395c7 from peer e7bcff2fef614e7a9981554c9e6db3c8 at {username='slave'} at 127.27.111.194:33611: session id = e7bcff2fef614e7a9981554c9e6db3c8-91db5c1b3d2f4d22b5882a61ea0395c7
I20240502 02:04:44.570006 7070 tablet_copy_source_session.cc:205] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:04:44.571044 27750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91db5c1b3d2f4d22b5882a61ea0395c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:04:44.573930 27750 tablet_copy_client.cc:783] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Starting download of 5 data blocks...
I20240502 02:04:44.580130 27750 tablet_copy_client.cc:647] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Starting download of 1 WAL segments...
I20240502 02:04:44.581597 27750 tablet_copy_client.cc:523] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:04:44.583968 27750 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 02:04:44.599293 27750 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:04:44.599625 27750 tablet_bootstrap.cc:492] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap complete.
I20240502 02:04:44.599825 27750 ts_tablet_manager.cc:1367] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20240502 02:04:44.600045 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:44.601796 27750 raft_consensus.cc:348] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:04:44.601928 27750 raft_consensus.cc:729] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 02:04:44.602058 27750 consensus_queue.cc:260] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:04:44.610605 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:44.612469 27750 ts_tablet_manager.cc:1398] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240502 02:04:44.613377 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session e7bcff2fef614e7a9981554c9e6db3c8-91db5c1b3d2f4d22b5882a61ea0395c7 received from {username='slave'} at 127.27.111.194:33611
I20240502 02:04:44.613453 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session e7bcff2fef614e7a9981554c9e6db3c8-91db5c1b3d2f4d22b5882a61ea0395c7 on tablet 91db5c1b3d2f4d22b5882a61ea0395c7 with peer e7bcff2fef614e7a9981554c9e6db3c8
I20240502 02:04:44.618083 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:44.627955 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:44.633251 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:44.635442 7050 raft_consensus.cc:1203] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 02:04:44.639364 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling UndoDeltaBlockGCOp(91db5c1b3d2f4d22b5882a61ea0395c7): 4921 bytes on disk
I20240502 02:04:44.639894 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: UndoDeltaBlockGCOp(91db5c1b3d2f4d22b5882a61ea0395c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240502 02:04:44.640607 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling UndoDeltaBlockGCOp(14e3c78c65334b52aca79d67c4bf45a2): 4834 bytes on disk
I20240502 02:04:44.640966 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: UndoDeltaBlockGCOp(14e3c78c65334b52aca79d67c4bf45a2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240502 02:04:44.714149 6985 connection.cc:535] client connection to 127.27.111.198:46475 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:04:44.715061 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:44.715144 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.715191 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.743400 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.748737 6718 connection.cc:535] client connection to 127.27.111.200:37235 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:04:44.748951 7114 connection.cc:535] client connection to 127.27.111.200:37235 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:04:44.749042 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 19 similar messages]
W20240502 02:04:44.749063 6849 connection.cc:535] client connection to 127.27.111.200:37235 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:04:44.749490 6982 connection.cc:535] client connection to 127.27.111.200:37235 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:04:44.749691 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:44.750706 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.750782 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.750946 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.751031 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.751081 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:44.751103 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.751127 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.751149 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 02:04:44.751813 6632 maintenance_manager.cc:419] P 8f0a4860df0c4e4a97901c87e80d6420: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7952 bytes on disk
I20240502 02:04:44.755146 6631 maintenance_manager.cc:643] P 8f0a4860df0c4e4a97901c87e80d6420: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.002s sys 0.000s Metrics: {"dirs.queue_time_us":152,"thread_start_us":67,"threads_started":1}
W20240502 02:04:44.757447 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.759374 6627 catalog_manager.cc:4573] TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 91db5c1b3d2f4d22b5882a61ea0395c7: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:44.765234 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:44.775631 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.778120 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:44.778851 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.797586 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.810526 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.830582 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:44.843681 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:44.856985 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 02:04:44.863024 6916 raft_consensus.cc:1203] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 02:04:44.907076 6783 raft_consensus.cc:1203] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240502 02:04:44.930272 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 02:04:44.976270 6916 raft_consensus.cc:1203] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.10->[3.11-3.11] Dedup: 3.11->[]
I20240502 02:04:45.020563 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:45.023801 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:45.025267 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:45.025822 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:45.026582 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:45.093086 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:45.278105 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:45.301961 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:45.326802 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:45.327553 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:45.329600 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:45.329710 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:45.341500 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:45.527495 27628 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:04:45.527674 27628 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:04:45.528201 27628 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:04:45.529322 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:45.529418 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:45.529495 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:45.529651 27628 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:04:45.530694 25747 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:04:45.530810 25747 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:04:45.531224 25747 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:04:45.532178 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:45.532275 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:45.532349 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:45.532480 25747 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 02:04:45.558432 27634 raft_consensus.cc:482] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:04:45.558650 27634 raft_consensus.cc:504] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 02:04:45.559166 27634 leader_election.cc:288] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
I20240502 02:04:45.559542 7182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66b932d884e4df8a78841103d88ded0" candidate_uuid: "c6101231591a458384f4ea6d1f0e0ae5" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "86bc8001c3464b929cd5c873861baa0c" is_pre_election: true
W20240502 02:04:45.560140 6985 leader_election.cc:334] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:45.560204 6985 leader_election.cc:302] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6101231591a458384f4ea6d1f0e0ae5; no voters: 4f53d80e39be410d92d4766692aff1c6, 86bc8001c3464b929cd5c873861baa0c
I20240502 02:04:45.560344 27634 raft_consensus.cc:2726] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:04:45.623793 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:45.640198 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:45.644909 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:45.647136 7161 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:04:45.660310 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:45.660409 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:45.735785 27628 raft_consensus.cc:482] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:04:45.735956 27628 raft_consensus.cc:504] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 02:04:45.736474 27628 leader_election.cc:288] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
W20240502 02:04:45.737672 7117 leader_election.cc:334] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:45.740049 7050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66b932d884e4df8a78841103d88ded0" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5" is_pre_election: true
I20240502 02:04:45.740206 7050 raft_consensus.cc:2454] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 2.
I20240502 02:04:45.740510 7117 leader_election.cc:302] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, c6101231591a458384f4ea6d1f0e0ae5; no voters: 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:45.740681 27628 raft_consensus.cc:2781] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 02:04:45.740746 27628 raft_consensus.cc:482] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:04:45.740783 27628 raft_consensus.cc:3037] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Advancing to term 3
I20240502 02:04:45.741642 27628 raft_consensus.cc:504] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 02:04:45.741989 27628 leader_election.cc:288] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 election: Requested vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 02:04:45.742584 7050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a66b932d884e4df8a78841103d88ded0" candidate_uuid: "86bc8001c3464b929cd5c873861baa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "c6101231591a458384f4ea6d1f0e0ae5"
I20240502 02:04:45.742722 7050 raft_consensus.cc:3037] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 FOLLOWER]: Advancing to term 3
W20240502 02:04:45.742935 7117 leader_election.cc:334] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:45.743611 7050 raft_consensus.cc:2454] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc8001c3464b929cd5c873861baa0c in term 3.
I20240502 02:04:45.743868 7117 leader_election.cc:302] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c, c6101231591a458384f4ea6d1f0e0ae5; no voters: 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:45.744026 27628 raft_consensus.cc:2781] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 02:04:45.744114 27628 raft_consensus.cc:686] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [term 3 LEADER]: Becoming Leader. State: Replica: 86bc8001c3464b929cd5c873861baa0c, State: Running, Role: LEADER
I20240502 02:04:45.744279 27628 consensus_queue.cc:237] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 02:04:45.746160 6636 catalog_manager.cc:5484] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: term changed from 2 to 3, leader changed from 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) to 86bc8001c3464b929cd5c873861baa0c (127.27.111.196). New cstate: current_term: 3 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
W20240502 02:04:45.758991 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:45.899940 27570 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:04:45.900072 27570 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:45.900507 27570 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:04:45.901463 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:45.902014 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:45.902087 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:45.902244 27570 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:04:45.914574 6627 catalog_manager.cc:4573] TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dcc5daef41bc44c4855f2bcb44e1c6d5: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:45.917635 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:45.962801 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:45.964035 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:45.965708 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:45.966065 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:45.968556 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:46.191395 7050 raft_consensus.cc:1261] T a66b932d884e4df8a78841103d88ded0 P c6101231591a458384f4ea6d1f0e0ae5 [term 3 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 02:04:46.192013 27628 consensus_queue.cc:1035] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 02:04:46.195574 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:04:46.202718 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:46.272555 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:46.273087 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:46.275144 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:46.275187 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:46.284451 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:46.321846 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:46.498710 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:46.574757 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:46.577785 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:46.577922 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:46.578497 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:46.588584 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:46.638041 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 02:04:46.799782 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:46.804765 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:46.887279 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:46.890204 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:46.892062 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:46.894114 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:46.902611 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:04:46.922008 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:46.928660 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:46.930073 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:46.966377 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:46.982092 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:47.003151 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:47.004443 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:47.037458 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:47.057799 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:47.086898 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:47.090605 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:47.106688 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:47.106846 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 02:04:47.203749 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:47.205971 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:47.206420 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:47.213558 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:47.215813 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:04:47.278445 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:47.310081 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:47.323261 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:47.327422 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:47.330911 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:47.378939 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:47.425662 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:47.438102 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 02:04:47.514989 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:47.516678 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:47.517119 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:47.518386 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:47.518481 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:47.530185 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 02:04:47.559957 27628 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:04:47.560096 27628 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:04:47.560498 27628 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:04:47.561543 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:47.561642 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:47.561705 7117 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:47.561831 27628 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:04:47.591079 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:47.722621 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:47.728408 25747 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:04:47.728538 25747 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:04:47.728942 25747 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:04:47.729900 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:47.729983 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:47.730070 8445 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:47.730204 25747 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:04:47.755461 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:47.816522 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:47.821660 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:47.823220 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:47.824316 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:47.829205 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:47.991149 6627 catalog_manager.cc:4573] TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dcc5daef41bc44c4855f2bcb44e1c6d5: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:04:48.029991 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:48.054649 28159 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:04:48.054797 28159 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:48.055267 28159 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:04:48.056079 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:48.056172 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:48.056241 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:48.056391 28159 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:04:48.084728 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240502 02:04:48.122361 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:48.123891 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:48.123979 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:48.124857 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:48.139416 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:48.172272 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:48.297371 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:48.429838 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:48.430859 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:48.431488 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:48.432897 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:48.444360 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:48.456722 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:48.603760 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:48.729328 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:48.733181 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:48.734571 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:48.738539 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:48.742473 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:48.767235 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:04:48.851542 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:48.968145 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 02:04:49.021551 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:49.035290 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:49.035578 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:49.037006 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:49.048434 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:49.050521 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:04:49.087643 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:49.264612 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:49.314111 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 02:04:49.338163 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:49.338992 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:49.339478 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:49.339779 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:49.341990 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:49.347007 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:49.378481 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:49.390728 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:49.412358 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:49.438441 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:49.448186 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:49.497395 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:49.522723 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:49.545605 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:49.613782 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:49.618000 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 02:04:49.646162 25747 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:04:49.646284 25747 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:04:49.646736 25747 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:04:49.647429 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:49.647521 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:49.647665 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:49.648526 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:49.648653 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:49.648697 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:49.648909 25747 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 02:04:49.660413 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:49.662112 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:49.662271 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:49.731676 28159 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:04:49.731871 28159 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:49.732316 28159 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:04:49.733402 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:49.733508 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:49.733568 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:49.733687 28159 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:04:49.741791 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:49.752444 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:49.779049 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 75 similar messages]
W20240502 02:04:49.779673 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:49.938725 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:49.959198 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 02:04:49.960883 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:49.961091 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:49.961345 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:49.962229 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:49.966720 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:04:50.025763 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:50.046770 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:50.070448 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240502 02:04:50.084223 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:50.132725 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:50.188302 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 02:04:50.262434 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:50.263406 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:50.267213 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:50.267561 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:50.269040 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:50.355525 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:50.482340 28335 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:04:50.482506 28335 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:04:50.482992 28335 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:04:50.483911 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:50.484004 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:50.484094 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:50.484218 28335 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:04:50.524230 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:50.526088 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 02:04:50.565070 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:50.565151 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:50.567816 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:50.570472 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:50.586541 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:50.604746 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:50.752523 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:50.871492 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:50.873888 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:50.874684 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:50.874945 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:50.877951 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:51.011456 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:51.170867 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:51.175434 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:51.180660 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:51.182832 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:51.183544 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:51.183614 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:51.184329 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:51.483662 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:51.488523 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:51.496512 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:51.502614 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:51.507866 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:51.590992 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:51.604841 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:51.674978 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 02:04:51.686146 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:51.718139 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:51.725100 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:51.777034 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 02:04:51.794432 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:51.797111 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:51.797349 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:51.800905 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:51.802593 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:51.834298 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:51.865449 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:51.877009 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:51.932252 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:51.937649 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:04:52.090605 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:52.094365 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:52.094707 6627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 116 similar messages]
I20240502 02:04:52.104333 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:52.105314 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:52.107594 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:52.107630 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:52.108007 6627 catalog_manager.cc:4573] TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dcc5daef41bc44c4855f2bcb44e1c6d5: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:04:52.114039 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:52.114670 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:52.123040 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:52.164287 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:52.167166 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:52.177181 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:52.194088 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:52.221571 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:52.251547 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:52.298444 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:52.318013 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 02:04:52.417574 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:52.419797 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:52.419911 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:52.420209 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:52.422582 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:52.573096 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:52.650719 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:52.715767 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:52.716977 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:52.719964 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:52.720639 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:52.725467 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:52.735428 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:52.775403 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:52.835075 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:52.982509 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 02:04:53.021291 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:53.023232 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:53.023747 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:53.025346 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:53.033598 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:04:53.143263 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 02:04:53.143903 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:53.176013 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:53.265012 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:53.319535 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:53.322818 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:53.324064 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:53.324448 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:53.338974 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:53.514760 25747 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:04:53.514915 25747 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:04:53.515362 25747 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:04:53.516252 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:53.516358 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:53.516438 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:53.516567 25747 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:04:53.554268 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:53.627620 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:53.628543 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:53.632107 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:53.633846 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:53.644106 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:53.766779 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:53.776540 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:04:53.777124 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:53.941491 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:53.941747 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:53.941963 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:53.943423 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:53.943966 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:54.015245 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:04:54.109063 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:54.181211 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 02:04:54.244400 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:54.244673 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:54.246362 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 02:04:54.247854 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:54.255390 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:54.257660 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:54.263801 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:04:54.285276 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:54.286638 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:54.304100 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:04:54.325837 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:04:54.461273 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:04:54.477191 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:04:54.478673 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 02:04:54.503748 28814 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:04:54.503903 28814 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:54.504354 28814 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:04:54.505383 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:54.505489 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:54.505568 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:54.505697 28814 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:04:54.554289 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:54.558322 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:54.560564 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:54.560787 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 02:04:54.567701 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:54.568780 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:54.593797 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:54.638543 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:04:54.679168 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:54.698354 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:04:54.743330 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:04:54.778647 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:04:54.790143 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:04:54.790943 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:54.802771 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 02:04:54.863570 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:54.868198 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:54.868800 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:54.871003 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 02:04:54.872085 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:54.879931 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:04:54.951529 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:04:54.979035 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:04:55.004539 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:55.165663 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:55.176762 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:55.178820 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:55.179255 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:55.180878 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:55.181618 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:04:55.199208 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240502 02:04:55.357791 29331 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:04:55.357941 29331 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:04:55.359133 29331 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:04:55.360071 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:55.360278 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:55.360363 7117 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:55.360502 29331 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:04:55.363862 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:55.418119 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 02:04:55.482249 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:55.482632 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:55.483794 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:55.483783 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:55.484058 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:55.499733 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:55.585011 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:04:55.686985 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:55.788844 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:55.789290 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:55.793879 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:55.800815 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:55.801178 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:55.933643 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:56.096567 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:56.101802 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:56.105227 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:56.106737 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:56.121760 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:56.125332 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:56.162110 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:56.306948 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 02:04:56.414878 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:56.415158 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:56.418177 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:56.423298 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:56.431155 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:56.500238 25747 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:04:56.500363 25747 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:04:56.500802 25747 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:04:56.501717 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:56.501819 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:56.501902 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:56.502029 25747 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:04:56.528033 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 02:04:56.543074 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:56.564653 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 02:04:56.649582 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:56.709304 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:56.724793 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:56.726480 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:56.726799 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:56.729894 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:56.735141 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:56.746325 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:04:56.747051 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 02:04:56.777082 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 02:04:56.805711 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 02:04:56.883920 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:04:56.952454 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 02:04:57.028800 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:57.029680 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:57.030262 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:57.031996 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:57.043578 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:57.111231 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 02:04:57.138823 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 02:04:57.160791 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:57.192817 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 02:04:57.207022 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 02:04:57.215972 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:57.221879 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:57.231333 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 02:04:57.297020 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 02:04:57.331890 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 02:04:57.333078 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:57.339962 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:57.342381 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:57.342559 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:57.350698 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:04:57.440270 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 02:04:57.517933 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 02:04:57.597440 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:57.626518 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 02:04:57.657833 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:57.657999 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:57.662647 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:57.665755 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:57.673743 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:57.749775 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:57.791257 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:57.823194 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 02:04:57.973968 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:57.976801 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:57.977078 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:57.978562 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:57.978950 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:58.019676 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:58.207132 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:58.233261 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 02:04:58.274473 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:58.275247 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:58.279225 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:58.289474 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:58.299269 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:58.359686 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:04:58.360329 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:58.458879 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:58.584164 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:58.584229 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:58.585391 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:58.587759 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:58.589243 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:58.714238 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:04:58.822292 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:58.844441 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:04:58.872715 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:58.887174 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:58.890900 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:58.891597 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:58.894277 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:58.905514 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:04:58.932318 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:58.965011 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:04:59.047101 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:04:59.166903 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 02:04:59.193645 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:59.196736 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:04:59.197063 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 02:04:59.200168 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:59.200451 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:59.212215 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:04:59.216560 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:04:59.228960 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:59.293571 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:04:59.330883 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:59.345988 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:04:59.352322 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:59.466230 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:04:59.499541 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 02:04:59.504062 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:59.505194 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:04:59.507954 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 02:04:59.514063 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:04:59.516053 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:59.520481 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:04:59.617767 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:04:59.648232 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:04:59.685752 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:04:59.710559 29880 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:04:59.710700 29880 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:04:59.711160 29880 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:04:59.712241 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:59.712334 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:04:59.712430 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:04:59.712553 29880 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:04:59.744745 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:04:59.777680 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:04:59.797286 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:04:59.801854 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:04:59.810433 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 02:04:59.814842 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:04:59.816674 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:04:59.823943 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:04:59.824550 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:04:59.829619 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:04:59.875466 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:04:59.892967 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 02:05:00.121248 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 02:05:00.122568 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:00.125885 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:00.125936 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:00.126999 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:00.133688 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:05:00.266918 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:00.293732 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:05:00.310873 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:00.345899 6627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 02:05:00.346652 6627 catalog_manager.cc:4573] TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dcc5daef41bc44c4855f2bcb44e1c6d5: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:05:00.356175 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:00.373194 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240502 02:05:00.426635 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:00.430555 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:00.432917 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:00.435621 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:00.449007 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:00.695163 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:00.737221 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:00.738323 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:00.738569 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:00.741876 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:00.767400 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:00.824316 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:05:00.853679 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:00.943027 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:01.046248 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:01.048931 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:01.051930 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:01.059319 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:01.063836 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:01.097770 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:01.139734 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:01.228183 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 02:05:01.354475 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:01.356556 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:01.357040 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:01.359310 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:01.366592 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:01.470638 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:01.486274 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 02:05:01.493863 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:01.500965 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:01.619182 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:01.641975 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:01.661835 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 02:05:01.669000 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:01.676777 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:01.678689 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:01.679102 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:01.688017 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:01.765880 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 02:05:01.819267 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:01.837574 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:01.837627 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:01.870766 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:01.981240 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:01.981452 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:01.985110 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:01.991144 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:01.992458 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:05:02.001982 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:02.010670 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:02.095940 30109 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:05:02.096082 30109 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:05:02.096491 30109 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:05:02.097602 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:02.097935 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:02.097999 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:02.098143 30109 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:05:02.135398 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:02.152062 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:02.169981 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:02.175683 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:02.280472 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 02:05:02.292805 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:02.293715 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:02.294126 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:02.295238 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:02.300626 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 02:05:02.313050 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:02.319739 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:02.342109 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:02.429526 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:02.466179 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:02.503472 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:02.602882 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:02.603305 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:02.606240 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:02.614645 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:02.616995 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:02.707180 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:02.713413 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:02.802459 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 02:05:02.913575 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:02.915745 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:02.916822 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:02.916875 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:02.918768 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:03.017279 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:03.042466 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:03.113974 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:03.141819 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:03.223542 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:03.225473 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:03.227360 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:03.229104 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:03.231591 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:03.450067 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:05:03.528674 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:03.528755 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:03.529510 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:03.534571 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:03.534835 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:03.544215 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:05:03.585546 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:03.684711 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:03.722275 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:03.793866 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:05:03.805238 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 02:05:03.830262 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:03.834640 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:03.834955 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:03.839999 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:03.851228 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:03.877888 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:05:03.921219 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 02:05:03.928109 30152 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:05:03.928244 30152 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:05:03.928699 30152 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:05:03.929709 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:03.929801 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:03.929888 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:03.930011 30152 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:05:04.066275 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:04.095826 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 02:05:04.097170 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:04.141402 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:04.144901 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:04.145481 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:04.150905 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:04.157444 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:04.175225 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:05:04.177017 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:05:04.307490 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:05:04.321276 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:04.322274 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:05:04.379252 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 02:05:04.459375 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:04.461138 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:04.461705 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:04.462270 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:04.470868 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:04.482534 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:04.576831 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:05:04.603256 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:05:04.623750 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:04.626237 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:05:04.643874 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:05:04.710453 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 02:05:04.761101 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:04.763896 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:04.767163 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:04.769927 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:04.774734 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:04.778182 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:05:04.804126 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:04.860896 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:05:04.871879 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:04.885722 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:05:04.918903 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240502 02:05:04.919564 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 02:05:05.067699 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:05.071242 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:05.071831 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:05.072038 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:05.073071 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:05.138015 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:05.352068 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:05.359263 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:05.385869 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:05.387326 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:05.390478 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:05.390744 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:05.395838 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:05.426743 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 02:05:05.635824 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:05.657650 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 02:05:05.670679 30413 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:05:05.670804 30413 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:05:05.671207 30413 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:05:05.672251 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:05.672343 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:05.672437 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:05.672590 30413 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:05:05.697485 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:05.698035 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:05.699163 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:05.700639 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:05.713726 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:05.783761 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:05.857817 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:05.995157 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:05.998805 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:05.999709 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:06.002559 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:06.002568 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:06.038709 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:06.141789 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:06.277056 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 02:05:06.301465 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:06.301476 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:06.301641 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:06.301977 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:06.304353 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:06.334249 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:06.334429 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:06.421654 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:06.456965 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:06.584228 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 02:05:06.588891 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:06.605363 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:06.605718 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:06.605856 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:06.607753 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 02:05:06.610271 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:06.612782 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:06.652287 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:06.700102 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:06.773497 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:05:06.781900 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:06.826174 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:06.863552 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:06.883344 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:06.907820 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:06.909826 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:06.912032 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:06.919432 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:06.919734 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:07.002209 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:07.002300 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:07.054100 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:07.097586 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:07.197724 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:07.211000 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:07.214385 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:07.214668 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:07.218674 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:07.227176 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:07.239286 30315 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:05:07.239403 30315 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:05:07.239765 30315 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:05:07.240873 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:07.241000 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:07.241044 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:07.241173 30315 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:05:07.250077 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:07.367030 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:07.371095 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:07.404971 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:07.497313 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:07.520020 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 02:05:07.520473 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:07.520895 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:07.521732 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:07.523741 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:07.531488 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:07.540153 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:07.583647 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:07.702991 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:07.836398 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:07.837272 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:07.839640 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:07.840934 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:07.844004 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:07.894737 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:08.120913 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:08.144234 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:08.144407 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:08.146076 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:08.146358 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:08.154830 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:08.285995 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 02:05:08.325340 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:08.392091 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:08.467396 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:08.467869 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:08.471683 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:08.475144 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:08.475533 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:08.578722 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:05:08.584614 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:08.608971 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:05:08.771996 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:08.772118 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:08.772644 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:05:08.783335 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 02:05:08.784374 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:08.784421 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:08.831758 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:05:08.855955 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:08.897938 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:05:08.940472 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:05:08.961961 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:05:08.986199 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:08.994767 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 02:05:09.068682 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:09.074319 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:05:09.074594 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 02:05:09.074702 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:09.076092 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:09.078971 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:09.079545 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:09.104769 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:09.243491 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:09.273165 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:05:09.274729 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:05:09.358085 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 02:05:09.382503 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:09.384656 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:09.385044 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:09.385388 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:09.385788 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:09.445470 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:05:09.504201 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:09.575369 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:09.575639 31283 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:05:09.575767 31283 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:05:09.576157 31283 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:05:09.577116 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:09.577212 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:09.577275 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:09.577407 31283 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:05:09.650943 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 02:05:09.689041 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:09.689514 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 02:05:09.692243 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:09.692386 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:09.693099 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:09.693498 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:09.769922 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:05:09.785791 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:05:09.822610 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:09.906245 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:05:09.939394 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 02:05:09.946714 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:09.973883 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 02:05:09.989616 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:09.991056 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:09.993002 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:09.995842 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:10.002019 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:10.041287 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:10.164775 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 02:05:10.292835 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:10.293789 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:10.293900 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:10.304155 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:10.304258 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:10.316825 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:10.368404 30315 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:05:10.368533 30315 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:05:10.368963 30315 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:05:10.369992 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:10.370098 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:10.370164 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:10.370293 30315 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:05:10.441574 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:10.441937 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:10.599236 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:10.599932 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:10.601261 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:10.604893 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:10.611706 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:05:10.627287 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240502 02:05:10.823087 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 02:05:10.835637 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:10.899673 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:10.904649 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:10.904696 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:10.907795 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:10.907979 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:10.908149 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:11.006974 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:11.031916 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:11.208063 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:11.212759 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:11.213425 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:11.214581 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:11.215209 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:11.351925 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:11.379855 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:11.417629 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:11.441334 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:11.464399 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:11.488382 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:11.490432 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:11.515389 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 02:05:11.516302 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:11.516458 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:11.517323 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:11.522182 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:11.531298 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 02:05:11.531411 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:11.595747 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:11.655357 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:11.705498 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:11.735107 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:11.755373 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 02:05:11.818708 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:11.821151 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:11.823065 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:11.831111 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:11.831696 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:11.882445 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:05:11.904268 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:11.918548 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:11.980932 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:12.038074 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 02:05:12.127455 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:12.128070 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:12.130882 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:12.131844 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:12.133309 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:12.143242 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:12.184659 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:12.251601 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:12.360888 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:12.384737 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:12.407735 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:12.410835 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:12.433182 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:12.436264 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:12.437711 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:12.439517 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:12.440107 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 02:05:12.447765 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:12.639613 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:12.647367 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:12.738899 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:12.739697 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:12.741345 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:12.741647 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:12.742725 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:05:12.905326 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:13.047958 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:13.048775 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:13.049353 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:13.052341 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:13.052691 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:13.239620 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:13.257686 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:13.284585 31283 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:05:13.284706 31283 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:05:13.285082 31283 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:05:13.285976 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:13.286067 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:13.286150 7117 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:13.286281 31283 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:05:13.352578 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:13.352741 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:13.356817 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:13.357282 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:13.363332 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:13.413262 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:13.489351 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 02:05:13.532012 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:13.561668 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:13.633756 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 02:05:13.657925 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:13.658020 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:13.663398 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:13.670682 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:13.675592 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:13.709380 31806 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:05:13.709499 31806 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:05:13.709836 31806 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:05:13.710774 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:13.710930 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:13.711019 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:13.711150 31806 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:05:13.720624 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:05:13.721438 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:13.855011 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:13.868690 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 02:05:13.963618 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:13.966991 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:13.967680 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:13.968238 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:13.970561 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 02:05:13.971355 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:13.972205 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:13.979676 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:14.044116 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:14.080623 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:14.086583 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:14.087126 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:14.144109 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:14.202196 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:14.237244 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:14.262892 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:14.264865 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:14.266110 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:14.276374 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:14.276962 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:14.335669 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:14.349207 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:14.394800 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:14.449383 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:14.518484 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 02:05:14.567420 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:14.570549 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:14.572770 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:14.577459 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:14.581807 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:14.683112 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:14.704419 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:14.757671 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:14.827847 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:14.829936 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 02:05:14.873908 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:14.875744 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:14.877537 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:14.878436 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:14.879313 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:14.910964 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:14.929044 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:14.962915 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 02:05:14.980849 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:15.185003 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:15.186236 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:15.190261 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:15.196645 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:15.204458 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:15.252925 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:15.461768 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:15.495251 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:15.497313 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:15.500171 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:15.510739 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:15.514552 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:15.596537 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:15.701965 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 02:05:15.802973 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:15.803989 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:15.804941 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:15.805999 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:15.807801 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:15.809399 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:15.907171 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:15.953550 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:16.082304 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 02:05:16.119256 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:16.120368 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:16.120360 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:16.121649 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:16.129335 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:16.132364 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:16.212406 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:16.303296 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 02:05:16.356288 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:16.385567 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:16.410372 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 02:05:16.426580 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:16.429361 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:16.429622 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:16.431886 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:16.432088 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:16.510058 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:16.660307 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:16.729168 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:16.729184 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:16.730607 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:16.731652 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:16.732584 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:16.742908 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:16.748519 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:16.750569 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:16.752421 6627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 02:05:16.760459 6627 catalog_manager.cc:4573] TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dcc5daef41bc44c4855f2bcb44e1c6d5: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:05:16.768090 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:16.811084 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:16.852593 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:16.898273 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:05:16.932324 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:16.965939 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:16.971887 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:16.988533 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 02:05:17.032630 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:17.034560 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:17.036243 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:17.036695 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:17.046591 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:17.048681 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:17.096653 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:17.134526 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:17.156307 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 02:05:17.200872 32024 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:05:17.200989 32024 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:05:17.201458 32024 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:05:17.202517 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:17.202617 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:17.202719 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:17.202857 32024 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:05:17.334446 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:17.337018 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:17.337399 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:17.341253 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:17.342566 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:17.406329 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:17.412503 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:17.466339 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:17.529810 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:17.600718 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:17.626569 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:17.648551 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:17.649395 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:17.650192 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:17.652056 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:17.665478 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:17.840655 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:17.956701 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:17.957921 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:17.958611 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:17.958715 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:17.960707 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:05:17.973421 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:18.105338 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:18.270246 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:18.271072 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:18.273809 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:18.273972 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:18.283381 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:05:18.352084 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:18.396915 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:18.509249 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 02:05:18.570515 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:18.572595 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:18.576967 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:18.577908 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:18.578835 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:18.696187 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:05:18.706972 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:18.778978 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 02:05:18.784876 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:18.816157 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 02:05:18.873500 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:18.875258 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:18.877008 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:18.878159 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:18.886574 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:18.902904 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:05:18.912111 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:05:18.913136 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:19.009781 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:05:19.085587 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:05:19.123765 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 02:05:19.183686 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:19.184864 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:19.184878 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:19.187019 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:19.196508 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:19.204161 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:19.317292 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:19.321313 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:05:19.393620 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:05:19.404181 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:19.429529 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 02:05:19.485922 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:19.489981 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:19.490057 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:19.499011 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:19.511421 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:19.540371 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:05:19.567132 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:05:19.574497 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:05:19.605268 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:05:19.617606 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:19.660063 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:05:19.768424 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 02:05:19.800451 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:19.800552 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:19.801852 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:19.803589 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:19.819680 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 02:05:19.824517 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:19.867563 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:19.940721 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:20.110414 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 02:05:20.118984 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:20.119092 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:20.120664 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:20.134605 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:20.143581 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:20.178025 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:20.193624 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:05:20.259095 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:20.276404 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 02:05:20.362865 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 02:05:20.435429 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:20.436161 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:20.437875 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:20.438856 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:20.453156 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:20.471185 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:20.715129 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:20.749666 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:20.750697 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:20.751173 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:20.751200 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:20.762559 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:20.852489 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:20.985080 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:21.045249 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 02:05:21.058038 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:21.059540 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:21.061972 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:21.062791 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:21.064947 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:21.185357 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 02:05:21.186141 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:21.269032 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:21.286444 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:21.364353 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:21.366376 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:21.369449 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:21.376394 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:21.384824 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 02:05:21.385957 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:21.397634 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:21.444881 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:21.448999 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:21.532725 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:21.589547 32724 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:05:21.589684 32724 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:05:21.590133 32724 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:05:21.591212 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:21.591318 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:21.591413 8445 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:21.591555 32724 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 02:05:21.679370 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:21.680065 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:21.681193 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:21.681514 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:21.697551 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:21.730021 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:21.751272 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:21.753394 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:21.914871 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:21.932363 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:21.936362 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:21.977366 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240502 02:05:21.978751 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 02:05:21.986722 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:21.988273 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:21.988865 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:21.998739 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:22.002557 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:22.035965 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:22.041319 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:22.058071 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:22.088375 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 02:05:22.103467 301 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:05:22.103633 301 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:05:22.104087 301 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:05:22.105144 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:22.105362 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:22.105419 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:22.105551 301 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:05:22.115867 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:22.143008 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:22.143260 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:22.214975 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:22.290383 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:22.293586 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:22.296029 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:22.297657 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:22.299485 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:05:22.330924 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:22.430332 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:22.473078 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 02:05:22.599289 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:22.600908 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:22.604709 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:22.606938 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:22.608669 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:22.639180 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:22.710623 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:22.767978 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 02:05:22.903342 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:22.903329 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:22.905968 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:22.906354 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:22.919142 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:23.012337 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:23.042281 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:23.173509 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:23.205246 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:23.211673 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:23.214124 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:23.214850 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:23.217950 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:23.225952 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:23.514508 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:23.523682 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:23.529289 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:23.531100 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:23.534886 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:23.535373 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:23.542636 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:23.645874 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:23.705955 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 02:05:23.730307 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:23.779315 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:05:23.780803 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:05:23.794562 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 02:05:23.844141 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:23.845963 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:23.852244 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 02:05:23.856719 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:23.857771 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:23.861261 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:24.074754 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:24.088059 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:05:24.126653 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:05:24.127298 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:24.158044 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:24.160749 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 02:05:24.164161 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:24.165215 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:24.167241 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:24.177690 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:24.275261 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:05:24.288087 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:05:24.342442 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:24.365823 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:05:24.430341 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 02:05:24.473624 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:24.476578 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:24.480404 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:24.482574 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:24.489002 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:24.511933 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:05:24.527454 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:05:24.554421 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:24.555099 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:05:24.613288 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:24.642143 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:05:24.669281 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:05:24.703850 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 02:05:24.785511 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:24.789467 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:24.790146 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:24.792644 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:24.800711 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:24.800936 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:24.953869 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:25.044677 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:25.097240 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:25.097671 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:25.106679 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:25.106959 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:25.113806 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:25.114583 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:05:25.158908 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:05:25.296828 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:05:25.329202 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:25.399120 992 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:05:25.399274 992 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:05:25.399698 992 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:05:25.400951 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:25.401114 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:25.401185 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:25.401407 992 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:05:25.415376 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:25.417585 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:25.418063 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:25.418257 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:25.430858 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:25.523715 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:25.578523 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:25.723392 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:25.725211 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:25.725808 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:25.727190 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:25.741292 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:25.765463 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:05:25.769171 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:25.908375 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 02:05:26.028414 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:26.030695 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:26.033179 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:26.034286 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:26.034524 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:26.035017 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:26.037585 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:26.062242 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:26.194768 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 02:05:26.249917 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:26.294953 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 02:05:26.335353 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:26.336406 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:26.342052 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:26.348963 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:26.349224 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:26.361528 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:26.413452 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:26.444536 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:26.477939 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:26.595921 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 02:05:26.640343 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:26.641886 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:26.642249 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:26.642900 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:26.650334 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:26.659551 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:26.739637 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:26.765290 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:26.771169 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:26.820528 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:26.827417 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:26.828601 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:26.835783 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:26.860062 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:26.933123 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 02:05:26.948058 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:26.948855 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:26.950223 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:26.952123 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:26.952616 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:26.973153 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:27.049525 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:27.065315 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 02:05:27.121495 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:27.212173 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:27.250823 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:27.253672 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:27.256108 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:27.257002 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:27.260061 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:27.381964 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:27.422418 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:27.560822 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:27.561291 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:27.562883 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:27.570786 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:27.571444 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:27.669081 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:27.821002 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:27.856002 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 02:05:27.880333 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:27.880708 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:27.882246 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:27.883946 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:27.893203 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:27.903894 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:28.002626 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:28.203908 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:28.204823 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:28.206084 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:28.206643 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:28.208019 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:28.215833 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:28.237020 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:28.250556 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:28.284933 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:28.471658 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:28.522330 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:28.524219 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:28.526592 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:28.534971 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:28.535291 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:28.604795 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:28.674870 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:28.719092 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:28.764847 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 02:05:28.787425 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:28.791941 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:28.836158 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:28.837694 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:28.838006 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:28.838989 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:28.853940 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:28.855998 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:28.883919 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:28.967835 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:29.085536 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 02:05:29.153656 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:29.155694 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:29.163513 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:29.163605 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:29.171267 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:29.221576 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:29.232811 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:05:29.233453 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:29.247177 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:05:29.247946 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:29.251039 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:29.362957 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:29.387015 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:29.410069 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:29.461078 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 02:05:29.471309 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:29.474042 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:29.474798 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:29.475072 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:29.478677 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:29.515627 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:29.521708 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:29.558832 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:29.621752 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 02:05:29.780504 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:29.783071 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:29.786558 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:29.789039 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:29.796787 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:29.848623 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:29.883594 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:29.887460 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:30.084671 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:30.089342 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:30.090222 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:30.090917 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:30.102912 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:30.248260 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 02:05:30.280153 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:30.320148 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:30.403074 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:30.406165 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:30.407063 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:30.407903 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:30.422976 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:30.428520 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:30.448880 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:30.481658 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:30.499604 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 02:05:30.713634 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:30.715780 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:30.717108 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:30.717522 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 02:05:30.722024 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:30.730566 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:30.766552 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:30.927744 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:30.936100 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:31.016711 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:31.018534 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:31.021912 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 02:05:31.022601 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:31.022601 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:31.022985 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:31.229378 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 02:05:31.260666 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:31.263110 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:31.290952 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:31.308851 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 02:05:31.309674 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 02:05:31.321786 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:31.325114 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:31.326143 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:31.328886 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:31.331496 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:05:31.391261 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:31.431218 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:31.484925 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:31.537711 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 02:05:31.598753 32724 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:05:31.598912 32724 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:05:31.599355 32724 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:05:31.600473 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:31.600555 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:31.600639 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:31.600767 32724 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:05:31.621412 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 02:05:31.632413 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:31.632577 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:31.642284 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:31.642879 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:31.643195 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:31.695173 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:31.805677 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:31.824579 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:31.824625 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:31.843871 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 02:05:31.935604 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:31.937289 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:31.945523 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:31.948335 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:31.948998 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:05:32.004722 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:32.037691 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:32.048452 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:32.103667 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:05:32.104310 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:32.109551 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:32.179092 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:32.200871 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:32.238982 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:32.240725 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:32.240890 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:32.241926 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:32.244503 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:32.256768 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:32.540246 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:32.546665 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:32.547200 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:32.551180 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:32.559346 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:32.566879 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:05:32.572629 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:32.659186 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:32.791827 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:32.858922 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:32.860096 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:32.860373 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:32.861986 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:32.869733 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:32.879681 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:32.921584 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:32.981729 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:33.140980 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:33.161198 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:33.168444 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:33.171388 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:33.177984 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:33.181280 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:33.320422 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:05:33.327536 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:33.476087 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:33.476189 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:33.478828 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:33.482589 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:33.487977 301 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:05:33.488109 301 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:05:33.488541 301 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
I20240502 02:05:33.489470 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:33.490221 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:33.491051 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:33.491130 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:33.491407 301 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:05:33.546510 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:33.558244 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:05:33.611964 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:33.674417 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 02:05:33.683228 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:33.727744 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:33.784574 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:33.786445 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:33.788543 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:33.790325 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:33.799544 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 02:05:33.804128 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:33.862951 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:05:33.891180 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:05:33.894112 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:05:33.993177 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:05:34.049659 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 02:05:34.088018 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:34.092587 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:34.095227 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:34.104432 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:34.104665 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:34.106009 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:34.106261 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:34.236150 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:05:34.289323 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:05:34.289937 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:05:34.320955 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:34.403573 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:34.403846 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:34.411653 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 02:05:34.413973 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:34.418241 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:34.418534 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:34.445117 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:05:34.495337 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:05:34.496666 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:34.505285 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:05:34.505899 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:34.509312 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:05:34.624353 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 02:05:34.710700 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:34.712044 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:34.712594 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:34.721464 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:34.721987 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:34.790184 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:34.844542 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:05:34.971053 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:35.014284 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:35.019249 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:35.019335 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:35.024606 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:35.038280 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:35.047524 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:35.166043 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:35.258051 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 02:05:35.323253 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:35.323551 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:35.324188 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:35.325330 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:35.326143 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:35.455502 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:35.489886 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:35.521291 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:05:35.559880 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 02:05:35.625648 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:35.626947 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:35.628793 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:35.629504 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:35.635031 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:35.715011 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:35.880153 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 02:05:35.927006 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:35.927114 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:35.930044 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:35.938041 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:35.939215 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:36.042717 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:36.066890 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:36.075428 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:36.103276 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 02:05:36.176146 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 02:05:36.225059 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:36.228268 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:36.228585 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:36.229302 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:36.243053 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:36.314312 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:36.326956 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:36.450953 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 02:05:36.451608 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:36.458750 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:36.460227 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:36.474615 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:36.531394 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:36.533622 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:36.539919 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:36.540956 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:36.546069 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:36.575065 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:36.647459 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:36.726035 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:36.771123 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:36.788884 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:36.813082 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 02:05:36.840927 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:36.841193 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:36.842242 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:36.846516 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:36.850252 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:36.897687 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:36.907310 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:36.961134 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:37.072003 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:37.093647 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 02:05:37.147274 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:37.148252 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:37.148550 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:37.151078 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:37.152765 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:37.164769 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:37.166111 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:37.193775 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:05:37.195549 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:37.385800 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:37.461315 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:37.461634 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:37.463295 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:37.464378 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:37.475453 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:37.493383 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:37.694562 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:37.770252 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:37.772315 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:37.772622 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:37.778584 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:37.780859 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:37.868170 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:37.937343 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:37.992542 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:38.026182 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 02:05:38.075239 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:38.075760 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:38.078001 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:38.078373 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:38.079197 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:38.126899 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:38.379951 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:38.380653 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:38.383898 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:38.386103 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:38.400771 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 02:05:38.407668 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:38.444895 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:38.476560 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:38.511656 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:38.523947 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:38.631484 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:38.696442 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:38.696713 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:38.701642 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:38.703056 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:38.713572 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:38.842294 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:38.849489 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 02:05:38.909835 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:38.922672 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:38.929569 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:38.973042 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:39.003823 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:39.004739 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:39.008111 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:39.008339 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:39.019730 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:39.077425 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:39.078984 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:39.118711 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:39.248024 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:39.297574 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:05:39.298516 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 02:05:39.312022 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:39.313277 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:39.315357 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:39.318054 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:39.321236 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:39.357322 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:39.379920 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:39.472103 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:39.500219 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:39.511389 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:39.546640 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:39.558544 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:05:39.559362 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:39.617303 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 02:05:39.631150 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:39.634521 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:39.634948 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:39.635237 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:39.636015 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:39.683563 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:39.687067 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:39.919808 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:39.952194 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:39.953119 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:39.955811 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:39.956108 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:39.962769 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:40.021740 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:40.152418 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:40.256371 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:40.257887 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:40.259840 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:40.265529 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240502 02:05:40.266708 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:40.268945 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:40.464253 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:40.472648 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:40.474293 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:40.539711 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 02:05:40.566465 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:40.566722 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:40.567366 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:40.571458 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:40.580474 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:40.594175 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:40.800338 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 02:05:40.877394 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:40.880404 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:40.880420 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:40.886237 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:40.889683 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:40.900869 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:40.922868 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:40.925860 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:05:40.967548 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:41.170759 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:41.188190 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:41.188417 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:41.188460 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:41.190502 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:41.193213 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:41.222702 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:05:41.335434 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 02:05:41.393075 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:41.471015 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 02:05:41.471915 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 02:05:41.495385 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:41.495663 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:41.496098 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:41.496413 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 02:05:41.496732 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:41.498382 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:41.510929 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:41.535038 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:05:41.616958 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:05:41.623183 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:05:41.643739 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:41.676185 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:05:41.769361 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:41.804412 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:41.808666 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:41.808763 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:41.811033 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:41.814646 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:05:41.856572 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:05:41.935595 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:41.977598 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:05:41.982877 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:05:42.057269 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:05:42.118141 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 02:05:42.119567 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:42.121397 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:42.123946 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:42.124409 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:42.138978 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:42.208873 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:05:42.223893 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:42.327150 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:42.333536 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:05:42.334152 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:05:42.348704 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:42.436532 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:42.436592 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:42.440460 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:42.441059 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:42.450946 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:42.718595 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:42.738608 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:42.741789 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:42.742218 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:42.750427 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:42.758618 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:05:42.857616 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 02:05:42.879703 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:42.923236 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:43.033764 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 02:05:43.045734 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:43.048183 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:43.049088 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:43.050637 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:43.052448 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:43.176625 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:43.309474 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:43.328377 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:43.334784 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 02:05:43.359256 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:43.359741 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:43.361892 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:43.363301 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:43.371268 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:43.533267 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:43.622058 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:43.629235 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 02:05:43.679191 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:43.681973 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:43.683359 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:43.683562 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:43.685746 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:43.878204 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:43.903688 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:43.938313 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:43.951375 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 02:05:43.966645 3400 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:05:43.966768 3400 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:05:43.967159 3400 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:05:43.968268 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:43.968595 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:43.968672 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:43.968812 3400 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:05:43.985392 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:43.986291 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:43.987988 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:43.990145 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 02:05:43.991071 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:43.991333 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:44.070161 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:44.102236 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:44.140437 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:44.230841 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:44.289773 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:44.289979 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:44.290477 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:44.290731 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:44.291116 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:44.338869 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:44.345252 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:05:44.369895 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:44.391358 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:44.394125 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:44.503291 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:44.555579 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:44.565065 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:44.568979 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 02:05:44.591917 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:44.594022 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:44.595860 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:44.595953 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:44.599643 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:44.686440 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:05:44.687122 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:44.696931 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:44.840166 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:44.870240 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:44.896327 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:44.900321 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:44.900918 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:44.903435 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:44.910748 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:45.107331 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:45.159605 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:45.200917 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:45.202769 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:45.203158 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:45.206303 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:45.212044 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:45.214670 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:45.323619 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:05:45.395628 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:45.505054 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:45.508224 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:45.510991 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:45.522280 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:45.530319 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 02:05:45.535055 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:45.606055 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:45.635368 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:45.637178 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:45.813712 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 02:05:45.826162 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:45.827445 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:45.835850 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:45.841229 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:45.845278 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:45.928787 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:05:45.952739 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:46.096863 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:46.142987 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:46.142987 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:46.143187 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 02:05:46.145606 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:46.151098 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:46.154184 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:46.157276 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:46.172035 2905 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:05:46.172147 2905 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:05:46.172526 2905 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:05:46.173555 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:46.174294 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:46.174361 8445 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:46.175187 2905 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:05:46.390100 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:05:46.400559 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 02:05:46.464792 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:46.466205 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:46.466351 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:46.467962 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:46.476049 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:46.496526 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:46.521165 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 02:05:46.584156 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 02:05:46.584751 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:05:46.605657 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:46.624553 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:05:46.652624 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:46.776201 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:46.776275 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:46.781069 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:46.785876 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 02:05:46.788005 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:46.800819 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:46.843608 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:05:46.905908 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:46.907310 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:05:46.951447 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:05:47.063127 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:05:47.065796 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 02:05:47.086199 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:47.087548 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:47.090617 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:47.092070 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 02:05:47.092186 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:47.101604 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 02:05:47.107152 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:47.197204 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:47.208227 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:47.242759 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:05:47.333058 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:47.339725 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 02:05:47.397390 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:47.399535 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:47.401966 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:47.402781 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:47.403059 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:47.586690 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:47.709844 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:47.709928 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:47.713651 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:47.722348 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:47.723070 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:47.754700 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:47.764112 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:47.975086 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 02:05:48.022279 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:48.033403 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:48.034511 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:48.034557 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:48.037057 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:48.059279 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:48.141993 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:48.171957 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:48.199786 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:05:48.242349 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:05:48.297699 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 02:05:48.331064 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:48.332281 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:48.334671 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:48.335266 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:48.342990 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:48.443013 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:48.464576 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:48.474676 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 02:05:48.632467 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:48.632596 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:48.635825 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:48.636196 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:48.647353 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:48.648432 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:48.794624 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:48.830163 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:48.913810 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 02:05:48.940248 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:48.941588 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:48.942221 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:48.943239 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:48.951527 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:49.010488 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:49.045794 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:49.053886 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:49.064236 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:49.072669 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 02:05:49.086258 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:49.172541 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:49.241173 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:49.242662 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:49.246244 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:49.247557 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:49.250286 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:49.392681 3731 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:05:49.392814 3731 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:05:49.393286 3731 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:05:49.394420 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:49.394501 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:49.394563 6849 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:49.394681 3731 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:05:49.440306 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:49.471910 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:05:49.472606 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:49.474812 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:49.521787 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:49.532575 6627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 02:05:49.534113 6627 catalog_manager.cc:4573] TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dcc5daef41bc44c4855f2bcb44e1c6d5: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
I20240502 02:05:49.543979 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:49.544844 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:49.546671 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:49.549108 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:49.551154 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:49.572307 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:49.611301 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:49.623926 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:49.642349 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:49.654881 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:49.789564 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:49.844436 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:05:49.850937 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 02:05:49.860823 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:49.861069 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:49.861181 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:49.862068 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:49.871827 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:49.900187 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:50.125730 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:50.147828 4445 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:05:50.147970 4445 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:05:50.148450 4445 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:05:50.150216 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:50.150462 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:50.150537 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:05:50.150688 4445 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:05:50.160961 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:50.165076 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:50.176988 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:50.180895 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:50.183259 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:50.228541 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:50.374624 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 02:05:50.477938 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:50.478765 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:50.482457 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:50.482545 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:50.493189 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:50.504490 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:50.507627 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:50.510254 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:50.528985 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:05:50.596982 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:50.618410 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:50.787801 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:50.789165 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:50.790809 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 02:05:50.791244 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:50.793814 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:50.799398 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:50.959967 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:51.000967 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:51.013028 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:51.024039 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:51.064527 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 02:05:51.094516 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:51.096225 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:51.097064 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:51.100342 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:51.118789 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:51.152835 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:51.341742 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 02:05:51.403777 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:51.405244 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:51.415179 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:51.415196 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:51.415819 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:51.421615 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:51.447533 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 02:05:51.447736 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:51.462520 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:51.487344 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:51.519290 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:51.675158 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 02:05:51.706296 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:51.708659 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:51.709313 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:51.709769 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:51.710557 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:51.832386 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:51.854454 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 02:05:51.933573 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:51.953997 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:51.989768 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:52.012202 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:52.013466 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:52.013954 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:52.013962 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:52.014732 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:52.130652 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:52.184509 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:52.194049 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:52.203905 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:52.227663 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:52.300840 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 02:05:52.313505 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:52.316977 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:52.317884 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:52.319360 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:52.327209 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:52.388708 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:52.425127 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:05:52.467803 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:52.485553 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:52.614272 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:52.618307 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:52.624316 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:52.625860 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:52.635022 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:52.661659 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:52.857901 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 02:05:52.918872 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:52.919579 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:52.922353 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:52.922529 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:52.922945 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:52.936295 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:52.947705 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:53.043280 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:53.074311 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:53.127497 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:05:53.222276 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:53.225239 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:53.227535 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:53.228492 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:53.236284 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:53.355953 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:53.388753 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:53.441174 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:53.536779 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:53.537005 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:53.538473 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:53.539392 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:53.539808 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:53.568177 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:53.658457 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:05:53.691395 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:53.838943 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:53.841308 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:53.841539 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:05:53.842008 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:53.846801 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:53.854804 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:53.870836 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:05:53.916059 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:53.929450 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:54.007463 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:05:54.044502 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 02:05:54.075657 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:05:54.150101 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:54.153230 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:54.153950 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:54.154436 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:54.164186 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:05:54.189144 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:05:54.248991 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:54.273948 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:05:54.295544 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:05:54.375025 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:54.456501 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:54.458557 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:54.459334 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:54.460217 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:54.462169 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:54.479283 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:54.481041 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:05:54.493779 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:05:54.553293 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:05:54.759503 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:54.759987 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:54.761907 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:05:54.764307 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:54.766543 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:54.767519 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:54.784633 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:05:54.862013 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:05:54.862622 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:54.876616 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:05:54.919648 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:54.934237 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:54.934590 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 02:05:55.062366 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:55.063242 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:55.066645 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:55.075317 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:55.076579 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:55.267199 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:05:55.333124 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:55.364164 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:55.366879 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:55.368827 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:55.371336 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:55.372155 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:55.395763 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:55.401875 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:05:55.531646 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:05:55.532316 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:55.678369 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:55.679702 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:55.681926 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:55.682772 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:55.684621 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:55.791394 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:05:55.811770 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:55.814597 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:05:55.898794 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:05:55.953255 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:55.979004 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:55.981341 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:55.982789 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:55.985208 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:55.997272 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:55.998694 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:56.194774 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:05:56.206604 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:05:56.247905 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 02:05:56.286885 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:56.289712 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:56.289906 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:56.289978 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:56.290846 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 02:05:56.304875 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:56.414857 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:56.444806 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:56.494894 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 02:05:56.518265 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:05:56.518725 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 02:05:56.592242 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:56.594128 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:56.594442 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:56.596813 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:56.598735 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:56.599447 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:56.672717 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:05:56.795656 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:05:56.855391 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:05:56.896433 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:56.898159 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:56.903928 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:56.903990 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:56.910713 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:56.934132 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 02:05:56.974234 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:56.999428 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:05:57.003140 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:05:57.081669 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:57.204555 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 02:05:57.208010 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:57.208478 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:57.209779 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:57.210913 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:57.211758 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:57.302467 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:57.389191 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:57.392382 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:05:57.498636 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 02:05:57.511567 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:57.513412 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:57.513468 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:57.513828 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:05:57.522426 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 02:05:57.539664 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:57.555922 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:05:57.576028 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:57.797140 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:05:57.804757 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:57.819806 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:57.820611 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:57.822266 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:57.823434 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:57.830565 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:57.851486 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:05:57.897750 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:57.993121 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:58.033763 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 02:05:58.129372 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:58.131866 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:58.132833 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:58.132856 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:58.135426 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:58.220269 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 02:05:58.322711 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:05:58.330551 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:58.333403 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:58.373373 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:58.434685 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:58.436404 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:58.436765 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:58.438114 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:58.445773 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:58.562083 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:05:58.627938 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:05:58.726168 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 02:05:58.729063 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:58.731068 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:58.733492 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:05:58.741256 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 02:05:58.742745 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:58.742986 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:58.762661 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:58.799482 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:58.829569 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:05:58.864274 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:58.955582 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:05:58.981830 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 02:05:59.027267 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:59.030473 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:59.031280 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:59.039510 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:59.039772 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:59.077011 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:05:59.182657 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:05:59.215653 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:59.248111 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:05:59.303418 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:59.320556 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:59.333359 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:59.335294 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:59.335563 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:59.337646 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:59.338073 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:59.632897 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:05:59.634094 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 02:05:59.640920 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:59.642356 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:59.643960 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:59.644807 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:05:59.650991 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:05:59.681723 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:05:59.762363 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:05:59.899389 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:05:59.914952 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:05:59.936508 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:05:59.946467 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:05:59.951129 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:05:59.952658 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:05:59.953328 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:05:59.953344 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:05:59.956681 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:05:59.987262 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:05:59.996762 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:06:00.087457 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:00.087893 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:06:00.142083 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:00.173951 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 02:06:00.255725 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:00.256420 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:00.257584 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:00.258890 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:00.258950 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:00.330677 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:00.436545 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:06:00.523128 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:00.539474 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240502 02:06:00.563860 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:00.564898 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:00.567332 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:00.570291 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:00.579699 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:00.706245 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:00.720100 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:06:00.825836 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:06:00.831034 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 02:06:00.872985 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:00.873703 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:00.878250 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:00.886936 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:00.887120 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:00.896859 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:00.910043 5170 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:06:00.910181 5170 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:06:00.910686 5170 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:06:00.914064 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:00.914160 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:00.914238 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:00.914373 5170 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:06:01.100003 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:01.117376 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:01.120587 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:06:01.155449 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:06:01.194435 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 02:06:01.194734 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:01.196259 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:01.198238 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:01.198660 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:01.210543 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:01.294909 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:06:01.324929 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 02:06:01.403210 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:01.448158 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:06:01.477506 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:01.494717 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:01.495631 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:01.495646 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:01.498817 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:01.505010 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:01.543545 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:06:01.643188 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:01.795646 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 02:06:01.797246 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:01.801637 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:01.802222 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:01.802913 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:01.804303 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:01.808909 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:01.856985 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:01.874184 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:06:01.976222 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 02:06:01.997982 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:06:02.077165 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 02:06:02.104120 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:02.104554 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:02.105648 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:02.124431 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:02.133157 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:02.203171 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:02.248451 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:02.380937 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:06:02.397310 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 02:06:02.424453 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:02.426370 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:02.426775 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:02.426900 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:02.427307 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:02.453176 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:06:02.456054 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:02.538975 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:06:02.572007 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:06:02.707003 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:02.711571 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 65 similar messages]
I20240502 02:06:02.728080 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:02.731184 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:02.732450 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:02.734782 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:02.752804 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:02.836792 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:02.892818 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 02:06:02.927361 2905 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:06:02.927498 2905 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:06:02.927958 2905 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:06:02.928910 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:02.929106 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:02.929167 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:02.929323 2905 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:06:02.968792 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:03.021834 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 02:06:03.035372 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:03.036525 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:03.037988 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:03.038486 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:03.040971 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:03.265368 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:03.336675 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:03.337144 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 02:06:03.344448 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:03.345980 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:03.346904 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:03.347621 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:03.355927 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:03.447873 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:03.468248 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:03.478988 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:03.656592 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:03.657115 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:03.664577 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:03.665503 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:03.666316 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:03.670148 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:03.736446 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:03.883771 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:03.884639 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 02:06:03.965214 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:03.966262 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:03.968561 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:03.974593 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:03.979537 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:03.993737 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:04.018303 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:04.023774 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 02:06:04.087544 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:04.089396 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:04.092178 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:04.219707 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:04.248126 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 02:06:04.273514 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:04.274432 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:04.274786 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:04.274842 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:04.277245 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:04.462905 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:04.561895 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:04.584043 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:04.585038 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:04.587086 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:04.588923 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:04.604550 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:04.609082 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:04.638579 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:06:04.639564 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:04.656837 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:04.809085 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:04.896068 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:04.899052 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:04.899369 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:04.900008 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:04.907711 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:04.964576 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:05.039999 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:06:05.040688 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:05.045255 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:05.063498 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:05.095973 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:05.114995 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 02:06:05.208098 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:05.208155 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:05.209491 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:05.209769 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:05.214567 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:05.223251 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:05.432359 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:05.441666 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:05.465160 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 02:06:05.479650 6313 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:06:05.479815 6313 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:06:05.480238 6313 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:06:05.481235 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:05.481338 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:05.481422 7117 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:05.481595 6313 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:06:05.512503 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:05.514340 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:05.516439 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:05.528625 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:05.538499 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:05.549856 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:05.590927 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 76 similar messages]
W20240502 02:06:05.772441 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:06:05.797021 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 02:06:05.823377 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:05.824307 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:05.836366 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:05.836664 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:05.837086 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:05.837716 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:05.884541 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:06:06.020252 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:06.047533 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:06.133958 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:06.135265 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:06.138325 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:06.138744 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:06.146916 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:06.287568 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:06.362162 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:06:06.408372 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:06:06.409911 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:06:06.424458 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:06.438448 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:06.439683 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:06.440619 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:06.442363 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:06.444128 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 02:06:06.446589 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:06.480693 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:06:06.534919 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 02:06:06.544975 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:06.640273 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:06:06.728339 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 02:06:06.741807 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:06.744660 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:06.745103 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:06.745908 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:06.750308 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:06.786953 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:06.870863 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:06.971863 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:06:07.012995 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 02:06:07.047088 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:07.047353 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:07.047906 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:07.049371 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:07.050565 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:07.161855 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:06:07.170046 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:07.184077 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:07.254021 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:06:07.262328 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:07.348423 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:07.350173 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:07.353379 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:07.354511 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:07.358404 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:07.377115 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 02:06:07.498023 7313 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:06:07.498174 7313 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:06:07.498646 7313 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:06:07.499667 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:07.499753 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:07.499845 6849 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:07.499985 7313 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:06:07.553301 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:07.620913 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:06:07.653123 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 02:06:07.655860 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:07.659716 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:07.666368 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:07.677052 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:07.677297 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:07.772711 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:07.774778 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:06:07.775352 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:06:07.777381 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:07.942946 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:06:07.959033 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 02:06:07.965052 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:07.966799 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:07.971257 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:07.971400 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:07.971949 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:08.060164 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:06:08.104902 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:06:08.133148 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:08.186203 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:08.193516 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:08.267537 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:08.268219 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:08.268674 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:08.269378 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:08.283265 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:08.513497 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:06:08.568428 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:08.571229 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:08.572340 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:08.574928 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:08.575911 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:08.583665 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:08.685089 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:08.780990 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:08.785384 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 02:06:08.815220 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:06:08.843641 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:06:08.855304 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 02:06:08.876459 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:08.879662 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:08.879985 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:08.886552 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:08.888012 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:09.045680 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:09.074231 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:06:09.087296 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:06:09.089284 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:09.183243 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 02:06:09.184417 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:09.186862 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:09.189599 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:09.191975 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:09.203213 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:09.248857 7313 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:06:09.248975 7313 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:06:09.249385 7313 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:06:09.250355 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:09.250466 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:09.250550 6849 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:09.250682 7313 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:06:09.254998 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:06:09.329502 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:09.469023 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 02:06:09.484163 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:09.485682 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:09.491333 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:09.492781 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:09.495318 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:09.532502 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:09.610409 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:06:09.659145 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:06:09.659907 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:09.803633 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:09.804587 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:09.804621 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 02:06:09.806120 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:09.806566 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:09.806972 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:09.810128 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:09.922657 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:06:09.932508 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:10.096845 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:06:10.112787 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 02:06:10.120558 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:10.120628 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:10.126137 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:10.128718 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:10.129092 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:10.144786 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:10.164996 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:10.280520 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:06:10.318944 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:10.333587 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:06:10.336191 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:10.347842 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:10.413633 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 02:06:10.429076 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:10.429642 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:10.430900 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:10.431105 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:10.434561 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:10.480758 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:06:10.624132 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:06:10.664682 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:10.729051 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:10.732393 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:10.733165 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:10.736145 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:10.736738 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:10.792419 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:10.883359 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:11.034427 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:11.036936 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:11.037333 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:11.038378 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:11.038812 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:11.046999 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:11.067476 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:11.188498 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:11.206035 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 02:06:11.238804 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:11.252339 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 02:06:11.339111 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:11.341377 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:11.342092 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:11.350026 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:11.352361 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:11.360155 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:11.414996 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:11.488183 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:11.492161 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:11.596422 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:11.644146 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:11.646889 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:11.648562 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:11.650418 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:11.657729 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:11.721382 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:11.748960 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:11.816613 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:11.940867 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:11.954156 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:11.955159 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:11.956235 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:11.957387 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:11.959455 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:12.013020 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:12.070706 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:12.182139 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 02:06:12.261937 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:12.262773 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:12.263087 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:12.265650 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:12.267586 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:12.281968 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 02:06:12.330938 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:12.433286 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:12.498064 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:12.537099 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:12.545478 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:06:12.548255 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 02:06:12.573329 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:12.573956 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:12.576136 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:12.576248 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:12.576499 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:12.579658 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:12.666044 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:12.717096 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:12.758904 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:12.803412 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:12.856879 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:06:12.872015 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 02:06:12.877092 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:12.877468 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:12.878285 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:12.879561 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:12.880594 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:13.086638 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:13.097759 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:13.184590 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:13.187776 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:13.189399 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:13.190874 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:13.198583 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:13.304841 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:13.315057 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:13.415017 6485 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:06:13.415171 6485 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:06:13.415642 6485 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:06:13.416848 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:13.416988 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:13.417040 8445 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:13.417207 6485 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:06:13.477066 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 02:06:13.490173 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:13.494071 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:13.495524 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:13.496492 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:13.510752 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:13.588860 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:06:13.601553 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:13.610074 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:06:13.625159 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 02:06:13.703845 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:13.799961 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:13.799961 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:13.800479 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:13.801679 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:13.804283 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:13.831434 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:13.885700 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:06:13.975032 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:06:14.001674 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 02:06:14.111997 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:14.112915 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:14.116235 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:14.118916 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:14.134660 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:14.144706 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:14.147537 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:14.266693 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:06:14.336475 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:14.426556 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:14.428210 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:14.428715 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:14.431592 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:14.433408 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 02:06:14.435951 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:14.539305 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:14.592501 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:06:14.670212 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 02:06:14.671029 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:14.706890 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 02:06:14.729512 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:14.733098 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:14.734162 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:14.734234 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:14.736640 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:14.763746 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:06:14.799096 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:14.883553 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:06:14.934201 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:14.974565 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:06:15.038947 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:06:15.041911 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:15.043458 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:15.044217 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:15.044459 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:15.046659 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:15.141270 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:06:15.157037 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:06:15.198711 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:06:15.238902 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:15.311766 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:06:15.312465 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:15.349422 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:15.350220 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:15.355324 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:15.358047 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:15.358309 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:15.384516 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:06:15.538538 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:15.620209 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:15.656854 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:15.662817 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:15.664347 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:15.664958 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:15.674804 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:15.729871 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:06:15.758642 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 02:06:15.890590 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:15.965016 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:15.967552 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 02:06:15.968619 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:15.968935 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:15.971545 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:15.978711 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:16.010131 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:06:16.019388 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:16.124161 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:16.147939 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:06:16.200206 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:06:16.263260 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:16.264937 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:16.266660 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:16.269907 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:16.283878 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:06:16.284580 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:16.381610 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:16.467219 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:06:16.571202 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:16.571961 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:16.572677 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:16.574309 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:16.584973 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:16.601390 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:16.632468 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:16.666244 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:06:16.758327 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:06:16.837165 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:16.873490 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:16.877352 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:16.877995 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:16.882520 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:16.891491 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:16.895378 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:06:17.064725 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:17.125656 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:17.178009 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:17.183070 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:17.190589 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:17.190737 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:17.191326 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:17.264199 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:06:17.279762 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:06:17.331228 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:06:17.387647 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:06:17.459421 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:17.460813 9115 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:06:17.460937 9115 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:06:17.461377 9115 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:06:17.464298 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:17.464393 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:17.464471 6849 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:17.464706 9115 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:06:17.480782 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 02:06:17.487332 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:17.488871 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:17.489414 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:17.491751 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:17.500562 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:17.515977 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:17.609627 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:06:17.640128 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:06:17.654629 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:06:17.660727 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:17.801807 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:06:17.806545 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:17.807271 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:17.807358 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:17.815402 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:17.816725 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:17.885183 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:17.909436 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:17.924811 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:06:18.081827 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 02:06:18.082620 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 02:06:18.114087 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:18.116690 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:18.119127 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:18.123910 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:18.138559 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:18.243821 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:18.339763 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:06:18.349370 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 02:06:18.412606 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:18.434687 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:18.435005 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:18.435396 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:18.438269 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:18.440973 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:18.465643 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:18.499190 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:06:18.736670 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:06:18.741427 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:18.743433 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:18.745716 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:18.746449 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:18.753791 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:18.759416 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:18.814901 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:06:18.867355 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:06:18.957304 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:19.011868 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:19.048401 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:19.048451 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:19.050338 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:06:19.051366 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:19.054421 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:19.058362 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:19.176330 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:06:19.337273 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:19.337409 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:06:19.348732 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:19.351650 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:19.352283 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:19.360817 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:19.361515 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:19.490582 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:06:19.511891 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:19.604503 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:19.669286 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:19.669581 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:19.670111 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:19.671613 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:19.679526 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:19.686535 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:06:19.756682 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:06:19.759404 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:06:19.783628 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:06:19.947669 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:19.961203 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:06:19.972851 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:19.972945 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:19.973502 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:19.974653 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:19.976330 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:20.003064 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:06:20.016017 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:20.170225 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:20.267417 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:06:20.272063 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:20.276615 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:20.276993 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:20.277673 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:20.277945 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:20.330255 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:06:20.396483 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:20.435303 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:06:20.437045 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:06:20.437589 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:20.488150 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:06:20.553351 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 02:06:20.576233 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:20.578218 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:20.579550 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:20.580098 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:20.581149 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:20.583036 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:20.643713 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:06:20.740664 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240502 02:06:20.741259 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 02:06:20.861277 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:06:20.872967 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:20.873167 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:20.874845 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:20.874876 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:20.876681 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:20.877128 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:21.008082 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:21.134016 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:21.168970 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:21.170953 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:21.174532 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:21.175412 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:21.188869 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:21.240893 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:06:21.251868 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:21.262818 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:06:21.375130 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:06:21.471464 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:21.474504 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:21.477627 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:21.477834 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:21.482659 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:21.483966 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:21.537477 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:21.657078 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:06:21.760138 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:06:21.778136 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:21.782797 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:21.783250 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:21.783479 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:21.786360 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:06:21.790736 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:21.873159 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:21.987500 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:06:22.055496 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:22.083086 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:22.084975 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:22.085595 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:22.086431 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:22.103544 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:22.104375 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:22.168357 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:06:22.397485 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:22.398015 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:22.399282 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:22.407294 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:22.407706 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:22.422931 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:22.423684 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:06:22.427623 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:06:22.475672 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:22.479883 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:22.485875 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:06:22.530468 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:06:22.536875 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 02:06:22.550165 9181 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:06:22.550289 9181 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:06:22.550698 9181 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:06:22.551115 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 02:06:22.551754 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:22.551838 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:22.551899 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:22.552023 9181 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 02:06:22.704202 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:22.705077 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:22.706111 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:22.706930 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:22.712332 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:22.715497 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:22.861585 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:22.936929 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:22.973304 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:06:22.983776 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:06:23.007669 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:06:23.010577 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:23.012789 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:23.013195 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:23.013468 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:23.021926 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:23.239734 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:23.311007 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:23.312812 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:23.313829 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:23.315281 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:23.322353 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:23.337033 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:23.357492 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 02:06:23.398054 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:06:23.405862 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:23.420998 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:23.554497 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 02:06:23.616439 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:23.628703 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:23.628736 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:23.631907 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:23.644423 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:23.659044 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:23.803577 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:23.812829 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:23.835043 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 02:06:23.931799 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:23.932632 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:23.933992 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:23.934890 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:23.940181 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 02:06:23.947033 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:23.981882 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:24.012228 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:24.017710 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:24.207130 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:24.236797 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:24.240464 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:24.240708 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:24.240952 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:24.243366 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:24.287113 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:24.382052 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:24.539788 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:24.540104 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:24.541404 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:24.546545 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:24.553486 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:24.553766 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:24.567876 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:24.663280 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:24.767776 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:06:24.775453 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:24.791870 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:24.796615 9773 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:06:24.796732 9773 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:06:24.797116 9773 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:06:24.797853 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:24.798275 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:24.798343 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:24.798503 9773 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:06:24.837416 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:24.842876 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 02:06:24.844235 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:24.845350 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:24.845782 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:24.847092 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:24.859082 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:24.931401 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:24.984216 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:25.036466 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:25.049180 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:25.152412 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:25.156387 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:25.158613 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:25.159047 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:25.160061 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:25.177321 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:25.274271 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:25.303040 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:25.329735 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:25.384444 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 02:06:25.438329 10308 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:06:25.438484 10308 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:06:25.438926 10308 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:06:25.439873 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:25.440331 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:25.440403 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:25.440546 10308 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:06:25.452457 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:06:25.453236 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:25.472694 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:25.477059 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:25.477149 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:25.477463 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:25.478616 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:25.479974 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:25.529600 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:25.685968 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:25.767928 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:06:25.768606 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:25.776763 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:25.778009 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:25.780489 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:25.786629 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:25.786890 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:25.913744 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:25.919036 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:25.943305 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 02:06:25.948628 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:26.081382 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:26.081885 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:26.082096 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:26.083526 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:26.086334 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:26.215014 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:26.266815 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:26.380859 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:26.385991 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:26.386044 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:26.387722 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:26.388183 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:26.389684 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:26.413559 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:26.434353 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:26.555550 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:26.604120 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:06:26.696758 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:26.699088 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:26.700420 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:26.701248 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:26.710651 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:26.750377 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:26.795094 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:26.855778 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:26.933768 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:26.988696 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:06:26.999662 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:27.001322 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:27.001595 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:27.004489 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:27.009285 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:27.009582 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:27.125722 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:27.234918 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:27.272078 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:27.302635 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:27.303383 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:27.303997 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:27.306496 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:27.308701 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:27.434480 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:27.435678 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:27.441226 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:27.462354 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:27.507215 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:27.511269 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:27.530671 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:06:27.606662 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:27.606652 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:27.608255 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:27.619382 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:27.630811 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:27.775784 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:27.787775 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:27.916338 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 02:06:27.917330 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:27.918730 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:27.919279 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:27.922693 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:27.934432 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:27.958364 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:27.963136 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:28.012133 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:28.107219 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 02:06:28.239805 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:28.242789 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:28.243337 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:28.244328 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:28.252218 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:28.253669 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:28.281049 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:28.373976 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:28.379007 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:28.397697 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 02:06:28.488909 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 65 similar messages]
I20240502 02:06:28.547955 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:28.548120 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:28.548411 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:28.558380 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:28.559888 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:28.563644 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:28.669852 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:28.706269 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:28.851801 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:28.852797 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:28.853448 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:28.859167 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:28.864569 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:28.865204 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:29.025349 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:06:29.070152 10308 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:06:29.070295 10308 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:06:29.070729 10308 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:06:29.071615 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:29.071713 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:29.071794 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:29.071919 10308 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:06:29.110309 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:29.116992 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:29.161595 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:29.162925 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:29.164235 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:29.170590 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:29.178167 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:29.208478 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:29.220729 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:29.264362 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:29.270681 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:29.391793 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 02:06:29.475121 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:29.476760 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:29.477500 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:29.481181 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:29.490140 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:29.495115 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:29.555825 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:29.560314 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:29.585067 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:29.714699 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:29.778143 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:29.779022 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:29.780225 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:29.782686 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:29.787948 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:29.845013 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:29.849380 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:29.901113 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:06:29.931277 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:06:29.984807 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:30.020589 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:06:30.084255 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:30.084694 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:30.085351 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:30.090581 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:30.112454 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:30.136623 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:30.241513 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:30.243305 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:30.246142 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:30.400981 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:30.401360 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:30.409853 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:30.413779 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:30.419194 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:30.444136 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:30.460637 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:30.556061 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:30.578047 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:30.683756 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:06:30.729096 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:30.732383 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:30.732574 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:30.736691 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:30.746225 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:30.746860 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:30.791139 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:06:30.791786 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:30.810859 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:30.839589 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:30.964229 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:06:31.039270 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:31.041278 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:31.043313 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:31.044766 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:31.045385 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:31.057854 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 02:06:31.218334 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:06:31.267872 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:31.342006 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:31.346357 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:31.348227 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:31.356794 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:31.359078 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:31.418927 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:31.457106 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:31.512630 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:06:31.609781 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:06:31.649897 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:31.652565 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:31.653831 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:31.656559 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:31.664367 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:31.731127 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:06:31.843520 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:06:31.853478 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:31.901358 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:31.963835 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:31.965318 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:31.967862 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:31.969151 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:31.974786 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:31.989212 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:32.035457 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:06:32.049599 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:06:32.069286 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:06:32.171267 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:06:32.176620 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:06:32.199143 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:06:32.204346 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:06:32.273181 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:32.276638 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:32.276930 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:32.286195 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:06:32.286981 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:32.290222 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:32.331805 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:32.371585 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:32.379947 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:32.587965 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:32.588735 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:32.589382 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:32.591127 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:32.591277 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:32.753551 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:32.784759 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:32.819495 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 02:06:32.826150 10308 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:06:32.826316 10308 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:06:32.826786 10308 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:06:32.827826 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:32.827920 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:32.828006 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:32.828147 10308 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:06:32.846710 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:06:32.882195 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:06:32.891096 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:32.894820 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:32.897826 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:32.900835 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:32.907438 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:32.907913 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:32.921247 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 02:06:33.186952 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:06:33.204175 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:33.210582 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:33.212487 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:33.212972 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:33.213318 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:33.223678 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:33.267704 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:06:33.404848 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:33.410430 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 02:06:33.434419 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:06:33.489356 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 02:06:33.498031 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:33.517791 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:33.521163 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:33.528905 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:33.533452 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:33.534565 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:33.619851 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:33.679523 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 02:06:33.833205 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:33.833997 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:33.836808 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:33.837888 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:33.857849 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:33.874038 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:33.945060 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:06:33.996980 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:34.012862 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:34.128929 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 02:06:34.144870 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:34.145531 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:34.147971 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:34.148267 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:34.160267 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:34.161794 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:06:34.279515 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:34.390301 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 02:06:34.449692 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:34.452275 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:34.453178 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:34.454919 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:34.464267 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:34.469796 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:34.494568 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:06:34.511351 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:34.524477 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:06:34.613045 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:06:34.677218 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 02:06:34.756758 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:34.758759 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:34.761065 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:34.761937 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 02:06:34.767963 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:34.783149 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:34.793772 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:06:34.810967 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:06:34.811005 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:06:34.834128 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:34.883239 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:35.036613 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:35.071033 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:35.071712 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:35.072002 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:35.072526 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:35.076061 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:35.134518 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:06:35.143158 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:06:35.335481 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:35.344375 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:35.381511 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:35.381871 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:35.383276 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:35.386181 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:35.386986 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:35.406463 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:06:35.614710 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:35.699267 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:35.699309 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:35.700016 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:35.701848 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:35.703272 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:35.735885 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:06:35.760074 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:06:35.786767 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 02:06:35.830606 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:06:35.858929 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:35.948542 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:35.957230 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 02:06:36.008720 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:36.014083 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:36.015506 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:36.015976 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:36.016026 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:36.070576 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:06:36.071314 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:36.076138 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:06:36.314909 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:36.316658 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:36.316718 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:36.318117 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:36.329581 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:36.434300 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:06:36.439476 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:06:36.469256 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:36.470022 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:36.563148 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:36.619832 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:36.620723 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:36.624437 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:36.630738 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:36.636657 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:36.684401 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:06:36.836439 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:06:36.840106 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:06:36.891139 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:36.929020 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:36.929606 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:36.930430 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:36.931598 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:36.933043 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:36.958592 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:37.026196 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:06:37.029829 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:37.180768 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:06:37.210958 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:06:37.219450 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:06:37.232831 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:37.235803 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:37.237246 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:37.240945 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:06:37.243532 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:37.246912 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:37.316802 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:06:37.385383 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:37.462219 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:37.480697 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:37.481269 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:37.483235 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:06:37.550233 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:37.551302 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:37.551546 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:37.554567 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:37.562801 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:37.566459 10561 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:06:37.566593 10561 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:06:37.567085 10561 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:06:37.568275 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:37.568379 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:37.568449 8445 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:37.568603 10561 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:06:37.684413 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:06:37.797621 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:06:37.860363 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:37.861846 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:37.865089 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:37.874282 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:37.874644 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:37.876377 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:37.978986 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:37.986037 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:38.078702 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 02:06:38.172399 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:06:38.175369 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:38.175366 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:38.177318 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:38.180260 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:38.192783 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:38.253613 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:06:38.413437 11961 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:06:38.413578 11961 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:06:38.414072 11961 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:06:38.415129 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:38.415213 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:38.415299 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:38.415448 11961 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:06:38.434805 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 02:06:38.467267 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:38.493371 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:38.496764 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:38.497463 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:38.500099 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:38.511031 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:38.540664 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240502 02:06:38.547302 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:38.550725 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:38.565392 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:38.567596 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 02:06:38.810977 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:38.811741 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:38.812040 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:38.813802 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:38.814657 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:38.891222 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:38.906297 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:39.001641 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:39.040679 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:39.102854 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:06:39.124032 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:39.129959 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:39.131107 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:39.131119 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:39.133311 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:39.178727 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:39.272054 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:39.392911 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:39.435554 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:39.435832 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:39.435988 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:39.436517 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:39.438694 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:39.445727 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:39.448895 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:39.544664 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:39.551241 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:39.631033 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:39.678014 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 02:06:39.740547 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:39.741497 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:06:39.743041 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:39.745956 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:39.748441 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:39.748508 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:39.845516 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:39.849766 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:39.878105 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:39.897636 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:39.944512 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:06:39.949612 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:39.960564 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:40.062019 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:40.063957 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:40.064162 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:40.064397 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:40.067155 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:40.067869 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:40.188316 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:06:40.299028 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:40.324165 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:06:40.375274 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:40.375481 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:40.376638 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:40.382915 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:40.383167 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:40.388751 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:40.674290 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:40.677058 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:40.684638 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:40.696050 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:40.696491 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:40.696832 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:40.702085 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:40.708194 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:40.856470 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:06:40.871264 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:40.878190 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 02:06:40.931344 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:41.000952 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:41.001147 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:41.005726 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:41.007217 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:41.015872 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 02:06:41.016219 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:41.068156 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:41.070047 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:06:41.163268 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:06:41.254154 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:41.310724 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:41.311990 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:41.316859 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:41.316952 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:41.317579 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:41.434262 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:06:41.436764 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:41.470583 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:06:41.573575 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 02:06:41.628353 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:41.629963 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:41.630190 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:41.636699 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:41.642802 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:41.645550 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:41.816196 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:06:41.877154 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:41.943529 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:41.946230 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:41.946765 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:41.948150 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:41.955430 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:41.961867 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:41.961973 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:06:41.988394 12050 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:06:41.988528 12050 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:06:41.988974 12050 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:06:41.990116 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:41.990262 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:41.990347 6849 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:41.990535 12050 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:06:42.103698 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:06:42.153628 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:42.211313 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:06:42.225203 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:06:42.236140 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 02:06:42.251228 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:42.253274 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:42.255268 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:42.256807 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:42.269104 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:42.341517 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:06:42.357260 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:06:42.361783 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:06:42.408356 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:42.468438 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:06:42.511668 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:42.562865 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 02:06:42.567867 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:42.569345 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:42.572170 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:42.576017 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:42.583775 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:42.712468 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:42.733381 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 02:06:42.880111 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:42.880429 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:42.886691 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:42.887540 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 02:06:42.889789 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:42.898532 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:42.972661 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:43.063762 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:43.183912 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 02:06:43.189564 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:43.202009 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:43.204535 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:43.206071 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:43.207408 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:43.210563 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:43.388780 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:43.432137 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:06:43.465320 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 02:06:43.513118 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:43.514097 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:43.522575 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:43.525353 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:43.536103 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:43.559953 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 02:06:43.576800 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:06:43.678447 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:43.681250 6632 maintenance_manager.cc:419] P 8f0a4860df0c4e4a97901c87e80d6420: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033339
I20240502 02:06:43.684418 6631 maintenance_manager.cc:643] P 8f0a4860df0c4e4a97901c87e80d6420: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.002s sys 0.000s Metrics: {"bytes_written":1851,"delete_count":0,"lbm_write_time_us":124,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":77,"threads_started":1,"update_count":25,"wal-append.queue_time_us":172}
I20240502 02:06:43.684834 6632 maintenance_manager.cc:419] P 8f0a4860df0c4e4a97901c87e80d6420: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.128008
I20240502 02:06:43.692270 6631 maintenance_manager.cc:643] P 8f0a4860df0c4e4a97901c87e80d6420: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":14446,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":331,"lbm_read_time_us":202,"lbm_reads_lt_1ms":17,"lbm_write_time_us":186,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":141,"threads_started":2,"update_count":25}
W20240502 02:06:43.713289 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:43.793803 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 02:06:43.824302 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:43.827311 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:43.830221 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:43.832252 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:43.833452 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:43.844944 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:43.898651 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 02:06:43.914458 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:06:44.004303 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:06:44.145835 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:44.147768 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:44.151458 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:44.155071 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:44.159183 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:44.230808 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:44.309764 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:44.315831 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:44.381878 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:06:44.458537 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:44.459599 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:44.460711 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:44.463321 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:44.465739 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:44.494951 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:44.566162 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:06:44.642848 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:06:44.756989 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:06:44.770213 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:44.770882 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:44.772667 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:44.773792 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:44.778566 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:44.784201 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:06:44.791384 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:06:44.793987 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:44.812707 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:44.835944 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:44.884429 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:44.927459 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:06:44.942992 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:06:44.993503 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:06:45.029824 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:06:45.077911 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:45.079361 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:45.079645 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:45.082628 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:45.083169 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:45.169421 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:06:45.220739 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 02:06:45.252818 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:45.296599 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:45.326781 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:45.386127 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:45.391212 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:45.392585 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:45.394255 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:45.395344 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:45.555176 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:06:45.563786 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 02:06:45.698325 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:45.699352 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:45.701639 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:45.705251 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:45.712442 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:45.721052 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:45.777287 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:45.785321 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:45.832607 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:45.964059 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 02:06:46.010622 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:46.012722 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:46.015152 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:46.018676 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:46.028102 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:46.223464 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:46.251747 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:46.276137 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:46.291358 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:46.326572 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:46.327630 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:46.327978 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:46.335795 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:46.345327 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:46.346848 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:46.499760 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 02:06:46.619110 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:46.623876 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 02:06:46.640780 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:46.644109 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:46.652050 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:46.653599 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:46.655901 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:46.722184 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:46.740103 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:46.787050 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:46.813177 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:46.831285 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 02:06:46.960999 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:46.962033 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:46.962491 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:46.964672 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:46.969640 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:47.068369 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:47.115769 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:47.168992 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:47.197183 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:47.208860 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:47.261935 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:47.262168 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 02:06:47.271054 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:47.272930 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:47.273203 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:47.279892 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:47.298789 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:47.348745 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:47.349399 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:47.408907 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 02:06:47.584687 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:47.585465 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:47.587616 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:47.591923 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:47.592446 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:47.593091 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:47.602878 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:47.624583 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:47.718361 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:47.750134 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:47.873317 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:47.873347 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 02:06:47.896776 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:47.897619 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:47.898283 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:47.899784 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:47.912335 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:47.972246 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:48.109398 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:48.145318 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 02:06:48.208899 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:48.212745 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:48.212750 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:48.214941 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:48.220649 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:48.268543 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 02:06:48.360229 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:48.522001 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:48.522505 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:48.525238 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:48.526280 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:48.534919 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:48.549655 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:48.651446 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:48.664652 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 02:06:48.831085 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:48.832229 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:48.834414 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:48.838569 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:48.840271 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:48.848258 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:48.880255 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:06:48.907260 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:49.122354 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:49.132391 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 02:06:49.146719 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:49.149797 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:49.149809 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:49.150131 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:49.153465 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:49.158340 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:49.158581 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:49.172262 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:49.232218 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 02:06:49.351706 13215 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:06:49.351881 13215 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:06:49.352305 13215 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:06:49.353471 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:49.353567 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:49.353629 7117 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:49.353788 13215 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:06:49.418474 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:49.464643 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:49.466174 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:49.466331 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:49.468878 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:49.483211 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:49.495481 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:49.524832 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:49.536778 6627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 02:06:49.537504 6627 catalog_manager.cc:4573] TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dcc5daef41bc44c4855f2bcb44e1c6d5: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:06:49.547966 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:49.555760 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:49.684633 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:49.689009 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:49.708283 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:49.724987 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:49.753897 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 02:06:49.770354 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:49.773267 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:49.776033 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:49.779561 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:49.779549 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:49.825795 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:49.892011 10561 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:06:49.892184 10561 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:06:49.892653 10561 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:06:49.893764 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:49.893867 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:49.893931 8445 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:06:49.894069 10561 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:06:49.941720 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:50.032274 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 02:06:50.080142 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:50.081538 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:50.083799 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:50.085547 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:50.106465 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:50.140969 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:50.163236 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:50.207235 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:50.209173 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:50.318557 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:50.328541 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 02:06:50.393410 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:50.394104 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:50.395632 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:50.398005 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:50.406843 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:50.407049 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:06:50.435446 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:06:50.556051 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:50.604512 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:50.657394 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 02:06:50.702895 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:50.702941 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:50.708096 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:50.710589 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:50.714572 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:50.753438 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:50.997231 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:51.006834 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 02:06:51.013988 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:51.015172 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:51.018018 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:51.024921 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:51.038193 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:51.073608 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 02:06:51.151684 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:51.163539 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:51.322324 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:51.325383 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:51.325700 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 02:06:51.326954 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:51.328539 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:51.336372 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:51.529078 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:51.543208 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:06:51.543818 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 02:06:51.559660 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:51.602558 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:51.631244 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:51.633390 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:51.633889 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:51.640101 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:51.641047 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:51.643883 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:06:51.717500 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:06:51.783886 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:06:51.907441 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 02:06:51.943023 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:51.943341 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:51.945725 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:51.948652 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:51.957576 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:51.960542 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:51.965268 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:06:51.972154 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:06:51.983311 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:06:52.030753 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:06:52.046871 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:52.047338 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:52.104800 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:06:52.169437 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:06:52.214071 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 02:06:52.248934 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:52.250178 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:52.253247 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:52.254911 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:52.262746 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:52.358513 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:06:52.456233 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:52.485391 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:52.504050 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 02:06:52.563328 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:52.563827 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:52.565213 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:52.566941 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:52.567134 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:52.644909 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:52.730566 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 02:06:52.872169 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:52.873404 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:52.873595 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:52.873733 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:52.874363 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:52.925691 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:52.981331 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:06:53.021456 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:53.022889 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:53.086675 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 02:06:53.182327 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:53.182368 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:53.184983 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:53.197851 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:53.197962 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:53.270346 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:06:53.408385 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:53.432739 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:53.495536 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:53.495821 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:53.497900 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:53.498543 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:53.499125 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:53.500074 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:53.549126 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:06:53.683279 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:06:53.770448 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 02:06:53.804900 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:53.804926 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:53.805891 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:53.810322 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:53.818615 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:53.936405 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:53.938922 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:53.969780 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:06:53.988711 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 02:06:54.020682 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:54.081715 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 02:06:54.117081 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:54.121023 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:54.121502 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:54.123167 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:54.133236 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:54.147043 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:06:54.156618 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:06:54.249640 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:06:54.299669 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 02:06:54.434676 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 02:06:54.434852 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:54.440414 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:54.442528 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:54.442965 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:54.444962 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 02:06:54.450721 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:54.453969 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:06:54.467924 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:54.469102 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:54.481124 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:54.507313 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:06:54.578343 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:54.628059 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:06:54.735579 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 02:06:54.749361 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:54.752252 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:54.752301 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:54.753793 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:54.755748 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:54.909297 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:06:54.942283 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:55.053345 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:55.054564 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:55.054632 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:55.057322 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:55.062590 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:55.083549 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:55.085004 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:55.105217 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:06:55.351014 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 02:06:55.364944 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:55.367790 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:55.372354 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:55.376740 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:55.382189 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:55.418587 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 02:06:55.435312 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:55.669348 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:55.679046 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:55.682425 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:55.683125 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:55.687848 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:55.687983 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:06:55.688122 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:55.690112 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:55.753648 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 02:06:55.898016 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:55.997041 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:56.001044 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:56.002825 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:56.006544 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:56.009773 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 02:06:56.013692 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:06:56.101979 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 02:06:56.102663 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:56.275259 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:56.293653 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:56.310179 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:56.311882 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:56.312274 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:56.313776 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:56.314006 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:56.420425 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:56.421919 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:56.485735 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:56.508549 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 02:06:56.616320 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:56.618651 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:56.620959 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:56.621779 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:56.629549 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:56.682359 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:06:56.687898 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:06:56.715435 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:56.809942 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:06:56.810711 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:56.833108 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:56.865161 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:56.868996 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:56.891662 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:56.895778 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:56.911804 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:56.929236 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:56.929940 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:56.931449 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:56.931777 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:56.933418 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:56.957428 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:57.100358 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:57.225772 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 02:06:57.236008 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:57.237489 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:57.238886 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:57.243647 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:57.251322 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:57.330139 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:57.330574 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:57.366842 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:57.405046 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:57.546936 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 02:06:57.547302 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:57.547719 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:57.547732 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:57.551348 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:57.552085 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:57.757730 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:57.849541 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:06:57.852442 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:57.855711 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:57.864276 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:57.866698 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:57.868697 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:06:57.891865 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:57.993541 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:58.021569 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:58.101343 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:06:58.147066 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 02:06:58.168113 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:58.171200 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:58.173867 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:58.176234 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:58.183519 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:58.215145 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:58.235284 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:58.381541 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:58.481618 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:58.481977 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:58.485430 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:58.486449 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:58.494716 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:58.563774 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:06:58.712504 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:58.718163 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:58.765558 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:06:58.790521 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:58.790589 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:58.790752 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:58.791467 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:58.792171 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:58.841452 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:06:58.849083 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:58.854765 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 02:06:58.966104 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:06:59.012534 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 02:06:59.093858 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:59.094419 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:59.094724 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:59.098018 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:59.106451 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:06:59.150380 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:06:59.171957 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:06:59.212466 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:59.261559 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:59.297446 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:06:59.330009 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:59.388193 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 02:06:59.405411 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:59.406877 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:06:59.409895 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:59.417439 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:59.425051 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:06:59.497467 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:06:59.505331 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:06:59.527254 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:06:59.590852 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:06:59.716003 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 02:06:59.722117 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:06:59.722267 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:06:59.722975 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:06:59.722976 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:06:59.725988 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:06:59.853080 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:59.853534 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 02:06:59.860041 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:06:59.869391 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:06:59.993965 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 02:07:00.028388 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:00.029076 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:00.031150 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:00.042165 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:00.044719 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:00.046689 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:00.241781 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:07:00.260726 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:07:00.284994 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:00.340770 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:00.341321 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:00.341358 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:00.342456 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:00.346967 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:00.395206 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:07:00.419701 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:00.482863 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:07:00.542749 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:00.642222 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:00.642566 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:00.644143 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:00.647027 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:00.657146 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:00.688058 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:00.765480 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:07:00.833341 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:07:00.920966 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:00.951108 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:00.952705 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:00.955178 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:00.955471 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:00.956918 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:01.057489 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:01.077201 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:07:01.154947 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 02:07:01.157613 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:01.220328 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 02:07:01.261216 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:01.261806 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:01.263010 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:01.265205 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:01.282279 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:01.323256 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:07:01.340449 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:01.388000 14787 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:07:01.388161 14787 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:07:01.388582 14787 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:07:01.389683 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:01.389802 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:01.389894 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:01.390040 14787 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:07:01.445922 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:01.494104 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:07:01.539561 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:01.546443 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:01.572177 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:01.573094 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:01.573776 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:01.575373 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:01.576090 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:01.811589 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:07:01.820259 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:07:01.820905 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:07:01.824615 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:01.833020 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:01.839442 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 02:07:01.881490 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:01.881945 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:01.882508 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:01.883339 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:01.890849 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:01.899772 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:07:01.912552 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:07:01.952908 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:01.999810 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:07:02.010236 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:07:02.138111 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 02:07:02.175237 15144 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:07:02.175397 15144 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:07:02.176019 15144 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:07:02.179184 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:02.179571 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:02.179634 7117 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:02.179903 15144 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:07:02.186769 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:02.189379 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:02.190101 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:02.199762 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:02.202913 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:02.329542 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:02.369010 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:02.468523 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 02:07:02.495021 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:02.497417 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:02.498935 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:02.499413 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:02.507413 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:02.569662 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:02.570827 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:07:02.604773 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:07:02.725275 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 02:07:02.801311 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:02.802467 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:02.806847 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:02.816701 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:02.819713 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:02.918960 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:02.961405 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:03.035203 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:03.091490 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 02:07:03.120939 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:03.124297 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:03.124848 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:03.125144 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:03.130575 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:03.153975 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:03.221024 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:07:03.365156 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:07:03.383699 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:03.416469 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:03.437448 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:03.441327 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:03.442548 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:03.444099 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:03.465504 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:03.507064 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 02:07:03.583122 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:03.752801 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:03.753384 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:03.755240 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:03.762604 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:03.765379 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:03.776827 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:03.936348 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:03.967550 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 02:07:03.968248 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:03.990818 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:04.045833 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:04.067772 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:04.069761 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:04.071959 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:04.072602 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:04.092159 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:04.099779 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:04.116633 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:04.173745 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:04.371527 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 02:07:04.383821 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:04.383909 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:04.384052 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:04.384790 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:04.397248 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:04.435034 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:04.435104 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:04.460381 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:04.479314 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:04.489151 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:04.498034 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:04.597834 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:04.692201 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:04.695583 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:04.696849 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:04.697474 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:04.699013 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:04.761700 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:04.783108 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:04.864200 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:04.923686 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 02:07:05.004276 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:05.005632 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:05.009490 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:05.014619 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:05.022320 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:05.037356 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 02:07:05.107777 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:05.129348 15803 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:07:05.129542 15803 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:07:05.129988 15803 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:07:05.131127 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:05.131217 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:05.131309 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:05.131443 15803 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:07:05.135334 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:05.184921 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:05.220438 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:05.312163 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:05.314932 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:05.315335 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:05.317126 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:05.318732 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:05.326803 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:05.449770 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:05.470882 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:05.575789 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:05.623833 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:07:05.631209 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:05.631402 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:05.632691 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:05.644479 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:05.644759 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:05.654633 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:05.800498 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:05.837054 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 02:07:05.838797 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:05.893481 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 02:07:05.948019 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:05.949256 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:05.951011 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:05.951324 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:05.953138 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:06.059047 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:06.108719 15757 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:07:06.108855 15757 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:07:06.109375 15757 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:07:06.110489 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:06.110702 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:06.110792 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:06.110952 15757 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 02:07:06.261862 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:06.264017 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:06.264462 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:06.265956 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:06.281106 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:06.351372 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 02:07:06.383770 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:06.399075 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:06.454262 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:06.526221 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:06.551626 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 02:07:06.573429 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:06.577651 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:06.579183 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:06.579234 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:06.579777 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:06.592113 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:06.647519 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:06.668790 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:06.670042 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:06.812151 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:06.836710 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:06.867331 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 02:07:06.883010 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:06.885671 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:06.888815 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:06.895874 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:06.896471 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:06.929729 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:07:06.930603 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:06.951889 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:07.020247 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:07.050971 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:07.081810 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:07.118608 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 02:07:07.197804 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:07.200192 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:07.204771 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:07.205425 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:07.214665 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:07.249069 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:07.265491 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:07.426748 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:07.443329 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:07.510915 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:07.512423 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:07.512959 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:07.514042 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:07.519474 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:07.528941 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:07.733186 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:07.788280 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:07.807413 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 02:07:07.822657 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:07.824671 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:07.824705 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:07.840024 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:07.843245 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:07.855773 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:07.970099 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:07.993367 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:08.031863 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:08.077107 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 02:07:08.148795 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:08.149313 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:08.150141 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:08.150696 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:08.151196 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:08.233359 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:08.329038 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 02:07:08.444051 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:08.473502 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:08.475075 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:08.478158 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:08.482530 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:08.483558 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:08.556509 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:08.753271 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:08.771106 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:08.783525 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:08.786696 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:08.787045 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:08.793692 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:08.795159 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:08.816998 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:08.946542 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:08.986902 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:09.022338 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:09.028290 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:09.061842 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 02:07:09.065008 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:09.095476 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:09.096305 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:09.098929 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:09.103650 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:09.111044 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:09.226372 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:09.282258 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:09.316980 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 02:07:09.403936 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:09.406282 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:09.409154 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:09.410804 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:09.424213 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:09.427004 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:09.504889 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:09.528350 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:09.531493 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:09.603546 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:09.620699 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:09.718776 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:09.720515 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:09.720744 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:09.732436 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:09.733013 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:09.802323 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:09.803699 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:09.869940 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:09.922792 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 02:07:10.032403 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:10.034487 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:10.037307 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:10.038575 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:10.050961 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:10.054955 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:10.090094 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:10.198123 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:10.227406 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 02:07:10.339749 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:10.343046 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:10.343825 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:10.350558 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:10.350919 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:10.415788 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 02:07:10.416590 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:10.464676 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:10.466866 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 02:07:10.547029 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:10.602886 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 02:07:10.645401 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:10.650790 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:10.651247 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:10.652740 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:10.652993 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:10.679876 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:07:10.680812 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:10.687075 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 02:07:10.959694 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:10.964939 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:10.965571 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:10.966068 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:10.976330 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:10.997987 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:11.125645 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:11.232815 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:11.294937 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:11.297818 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:11.302083 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:11.302342 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:11.303726 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:11.399022 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:07:11.415663 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:11.420053 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:11.430550 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:11.516453 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:11.544996 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:11.594061 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:11.607129 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:11.610872 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:11.619114 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:11.619645 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:11.621407 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:11.698092 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:11.751497 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:11.759462 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:11.801533 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:11.826547 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 02:07:11.937930 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:11.941411 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:11.942721 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:11.943711 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:11.963160 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:12.013475 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:12.061383 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:12.104411 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:12.151804 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:12.163236 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:07:12.163914 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:12.205212 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:12.247866 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:12.248266 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:12.249208 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:12.254571 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:12.265578 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:12.297330 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:12.475251 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 02:07:12.555986 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:12.556547 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:12.557471 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:12.559695 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:12.568307 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:12.603101 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:12.636745 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:12.711871 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:12.712320 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:12.718221 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:12.737340 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:12.860391 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:12.861269 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:12.872946 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:12.873929 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:12.886138 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:12.924320 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:13.000334 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:13.086859 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 02:07:13.172888 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:13.173727 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:13.182015 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:13.191192 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:13.191412 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:13.200345 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:13.266031 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:13.279559 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 02:07:13.283739 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:13.487655 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:13.493433 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:13.494166 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:13.496685 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:13.505307 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:13.680045 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:13.727149 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:13.731928 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:13.797957 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:13.798527 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:13.801777 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:13.811407 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:13.812275 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:13.866880 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:13.934796 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:13.935606 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:14.083329 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:14.100057 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:14.100298 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:14.100545 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:14.103724 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:14.123726 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:14.124871 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:14.148903 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:14.157827 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:07:14.201762 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:14.207226 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:14.259135 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:14.287393 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 02:07:14.334260 16677 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:07:14.334434 16677 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:07:14.334864 16677 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:07:14.335999 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:14.336082 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:14.336135 7117 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:14.336270 16677 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:07:14.377957 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 02:07:14.412806 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:14.412856 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:14.422116 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:14.422125 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:14.423964 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:14.480270 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:14.517141 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:14.538167 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:14.584029 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:14.606288 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:14.671777 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:14.720121 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:14.722976 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:14.725033 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:14.732897 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:14.733959 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:14.811908 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:14.868909 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:14.977773 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:15.028975 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:15.029505 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:15.029495 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:15.039005 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:15.040886 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:15.182680 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:15.216706 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:15.237854 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:15.307237 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 02:07:15.328243 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:15.329702 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:15.335633 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:15.339041 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:15.346669 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:15.381490 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:15.470707 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:15.474123 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 02:07:15.474876 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:15.497586 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 02:07:15.529579 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:15.639834 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:15.640695 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:15.641032 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:15.642884 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:15.642939 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:15.643153 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 02:07:15.650801 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:15.692332 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:15.723424 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 02:07:15.945536 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:15.947286 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:15.948689 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:15.952973 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:15.954272 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:16.058849 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:16.060609 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:16.203866 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:16.257589 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:16.257686 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:16.258764 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:16.260571 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:16.263047 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:16.279881 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:16.341161 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:16.412930 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:07:16.465811 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:16.547464 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:16.555783 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:16.556339 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:16.557193 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:16.558507 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:16.566706 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:16.598027 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:16.600003 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:16.602975 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:16.675038 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:16.676975 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:16.744010 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 02:07:16.855863 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:16.856465 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:16.858224 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:16.865833 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:16.868819 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:16.929244 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:16.996330 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:17.017055 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:17.095650 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:17.129887 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:07:17.144737 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:17.165891 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:17.166496 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:17.170957 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:17.171329 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:17.178200 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:17.214516 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:17.353834 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:17.393206 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:07:17.393762 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:17.414997 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 02:07:17.475260 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:17.476112 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:17.476683 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:17.489586 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:17.489625 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:17.537647 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:17.574961 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:17.671005 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:17.711252 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:17.731649 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 02:07:17.777261 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:17.782249 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:17.782449 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:17.787402 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:17.794700 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:17.822232 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 02:07:17.974015 16398 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:07:17.974159 16398 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:07:17.974629 16398 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:07:17.975471 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:17.975587 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:17.975664 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:17.975788 16398 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:07:17.993006 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 02:07:18.012881 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:18.084055 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:18.084952 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:18.086278 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:18.094744 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:18.095592 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:18.151358 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:18.152683 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:07:18.161551 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:18.206691 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 02:07:18.393225 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:18.394780 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:18.394872 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:18.395807 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:18.398571 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:18.547572 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:18.554724 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:18.699899 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:18.703131 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:18.703953 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:18.707818 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:18.711807 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:18.754161 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:18.783764 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:07:18.803681 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:07:18.831151 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:07:18.985304 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 02:07:19.003629 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:19.004932 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:19.005004 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:19.009820 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:19.011142 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:19.065737 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:19.082870 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:07:19.117846 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:07:19.124199 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:19.169818 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:19.210796 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 02:07:19.304457 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:19.306567 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:19.307471 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:19.308495 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:19.308652 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:19.353397 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:19.388263 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:07:19.540731 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:07:19.544271 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:19.551488 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:19.609136 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:19.613518 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:19.614001 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:19.614595 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:19.617769 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:19.623561 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:19.747716 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:07:19.835876 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:19.896544 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 02:07:19.915099 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:19.916610 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:19.916714 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:19.917117 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 02:07:19.917541 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:19.918785 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:19.936363 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:20.035188 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:20.084167 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:07:20.118257 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:20.167490 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 02:07:20.216725 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:20.219537 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:20.219619 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:20.220571 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:20.230487 16677 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:07:20.230600 16677 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:07:20.230954 16677 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
I20240502 02:07:20.233130 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:20.233863 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:20.234915 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:20.234983 7117 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:20.235245 16677 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:07:20.279438 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:07:20.331718 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:07:20.390309 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:20.498085 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:20.522639 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:20.523367 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:20.524353 16020 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:07:20.524459 16020 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:07:20.524847 16020 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:07:20.527868 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:20.531035 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:20.532488 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:20.532559 6849 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:20.532781 16020 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:07:20.534704 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:20.535615 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:20.573212 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:20.585726 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 02:07:20.586297 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:20.674255 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 02:07:20.715284 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 02:07:20.827379 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:20.827922 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:20.833844 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:20.834856 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:07:20.836143 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:20.836946 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:20.837481 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:20.999480 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:21.001904 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:21.137751 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:21.139132 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:21.140188 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:21.147491 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:21.147797 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:21.239099 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:21.391212 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:21.400650 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 02:07:21.454089 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:21.454641 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:21.455085 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:21.455202 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:21.459240 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:21.492551 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 02:07:21.493242 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:21.569103 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:21.569094 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:21.604380 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:21.696578 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:21.718829 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:21.721848 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 02:07:21.758383 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:21.758842 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:21.770563 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:21.771708 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:21.771798 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:21.856426 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:21.891626 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:22.062260 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:22.063032 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:22.070541 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:22.071092 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:22.071544 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:22.087747 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:22.110186 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:22.257437 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:22.268080 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:22.293172 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:07:22.293507 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:22.370249 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:22.373042 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:22.373420 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:22.375487 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:22.379680 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:22.396975 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:22.449172 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:22.458935 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:22.475198 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:22.488014 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:22.518131 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:07:22.545359 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:22.680733 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:22.684370 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:22.694170 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:22.694823 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:22.696818 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:22.711503 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:22.715828 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:22.853713 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:22.949054 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:22.990362 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:22.992708 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:22.992846 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:22.993434 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:23.006043 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:23.016191 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:23.073876 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:23.080864 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:07:23.282538 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:23.294998 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:23.296644 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:23.297688 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:23.300010 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:23.310142 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:23.343327 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:23.376091 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 02:07:23.459450 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:23.596789 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:23.597240 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:23.597839 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:23.600311 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:23.614791 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:23.781078 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:07:23.792294 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:23.870581 16398 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:07:23.870759 16398 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:07:23.871176 16398 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:07:23.872309 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:23.872418 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:23.872504 8445 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:23.872642 16398 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:07:23.890595 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 02:07:23.908425 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:23.908703 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:23.913007 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:23.920868 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:23.926595 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:23.937575 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:23.970454 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:24.004261 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:07:24.011808 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:07:24.157663 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:07:24.169806 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:07:24.197994 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:24.218670 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 02:07:24.219863 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:24.220007 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:24.220431 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:24.220775 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:24.236999 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:24.254849 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:07:24.484589 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:07:24.509181 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:24.523977 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:24.526100 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:24.526356 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:24.540390 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:24.546622 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:24.557283 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:24.580152 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:24.760831 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 02:07:24.844725 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:24.846165 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:24.847293 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:24.851523 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:07:24.858950 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 02:07:24.861518 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:24.864616 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:24.896268 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:07:24.923661 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:07:24.924822 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:07:24.960544 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:25.131906 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 02:07:25.156302 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:25.160107 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:25.160271 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:25.161046 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:25.161831 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:25.173734 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:25.203639 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:25.352190 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:25.354460 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:07:25.392880 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 02:07:25.464018 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:25.464663 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:25.466321 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:25.467746 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:25.468505 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:25.514284 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:07:25.517194 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:07:25.574637 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 02:07:25.609073 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:25.612677 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:25.737995 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:25.766530 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:25.767684 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:25.769984 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:25.775978 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:25.779556 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:25.852458 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:07:26.068708 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:26.069361 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:26.071648 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:26.072539 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:26.075275 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:26.075918 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 02:07:26.080255 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:26.107431 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 02:07:26.128263 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:26.216682 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:26.371402 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:26.373030 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:26.374812 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:26.374903 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:26.375455 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:26.466989 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:07:26.547662 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:07:26.560812 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 02:07:26.615887 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:26.623888 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:26.632647 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:07:26.659703 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 02:07:26.672370 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:26.676110 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:26.681565 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:26.682515 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:26.696602 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 02:07:26.704414 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:26.749213 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:26.830448 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:07:26.848114 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 02:07:26.982626 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:26.984567 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:26.987813 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:26.988924 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:26.997695 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:27.107919 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:07:27.112396 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:27.163419 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:07:27.179970 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:27.207965 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:07:27.224619 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 02:07:27.288528 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:27.289726 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:27.289981 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:27.292423 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:27.293869 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:27.330999 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:27.400597 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:07:27.446671 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 02:07:27.596076 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:27.597807 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:27.602876 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:27.608366 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:27.610190 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:07:27.612879 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:27.618044 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:27.619139 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:07:27.736505 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:27.743516 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:07:27.857616 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:27.905261 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:27.906811 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:27.906843 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:27.909300 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:27.926705 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 02:07:27.927184 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:28.049937 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:07:28.059823 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:07:28.156934 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:28.209419 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:28.215075 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:28.217545 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:28.219872 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:28.222465 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:28.223590 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:28.250106 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 02:07:28.379838 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:28.425395 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 02:07:28.530777 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:28.533960 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:28.544953 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:28.545295 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:28.547246 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:28.581997 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:28.701514 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:28.851027 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:28.851541 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:28.853938 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:28.856343 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:28.864234 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:28.867655 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:28.868546 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:28.926072 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:07:28.975282 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:07:29.032586 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:07:29.102694 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 02:07:29.158078 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:29.162434 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:29.164117 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:29.166538 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:29.173805 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:29.198457 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:29.249958 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:07:29.282431 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:29.310026 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:07:29.379130 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:07:29.420403 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:07:29.443632 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:29.471813 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:29.471856 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:29.473760 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:29.475795 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:29.486809 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:29.528038 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:29.630564 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:29.706266 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:07:29.712118 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:29.769346 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 02:07:29.773427 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:29.773623 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:29.775230 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:29.777439 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:29.779299 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:29.901583 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:29.914048 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 02:07:30.074131 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:30.074679 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:30.076261 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:30.076810 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:30.077419 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 02:07:30.079361 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:30.106266 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:07:30.171551 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:30.172389 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:30.205929 16398 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:07:30.206082 16398 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:07:30.206540 16398 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:07:30.207602 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:30.207703 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:30.207782 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:30.207938 16398 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:07:30.345203 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:07:30.357612 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 02:07:30.384722 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:30.385352 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:30.386961 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:30.388185 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:30.394912 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:30.478161 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:30.502523 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:07:30.551331 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 02:07:30.580994 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:30.653126 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:30.683889 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 02:07:30.693298 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:30.699656 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:30.699896 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:30.701462 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:30.701565 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:30.896521 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:07:30.939340 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:30.998687 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:30.999078 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:30.999400 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:31.000456 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:31.011111 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:31.020709 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:31.046931 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:31.178088 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:31.180380 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:31.257040 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 02:07:31.263065 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 02:07:31.303550 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:31.303557 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:31.304340 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:31.315757 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:31.323370 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:31.438503 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:31.455757 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:31.506237 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:31.517966 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:31.547703 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:31.566570 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:31.588851 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:07:31.589557 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 02:07:31.614464 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:31.616585 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:31.617951 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:31.619390 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:31.627007 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:31.709484 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:31.719224 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:31.922677 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 02:07:31.923368 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:31.925951 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:31.925948 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:31.925990 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:31.934617 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:31.951928 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:32.049791 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:32.071386 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:32.230311 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:32.230546 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:32.230998 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:32.232331 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:32.234678 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:32.243441 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:32.268263 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:32.280119 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:32.409320 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:32.499060 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:32.535908 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:32.539633 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:32.543179 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:32.544342 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:32.552517 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:32.560868 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:32.565611 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:32.724831 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:32.801054 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 02:07:32.847535 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:32.854601 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:32.856175 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:32.857120 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:32.860221 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:32.890457 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:32.968119 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:32.991291 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:32.997547 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:33.032050 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:07:33.032616 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:33.057444 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 02:07:33.062276 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:33.158063 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:33.159250 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:33.160974 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:33.163591 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:33.185428 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:33.416363 19555 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:07:33.416491 19555 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:07:33.416939 19555 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:07:33.416939 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:33.417847 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:33.418100 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:33.418171 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:33.418315 19555 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:07:33.449420 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:33.480543 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:33.480896 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:33.481060 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:33.481379 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:33.481752 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:33.484457 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:33.569798 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:33.584031 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:33.625340 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 02:07:33.786056 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:33.786819 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:33.787542 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:33.790477 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:33.801795 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:33.916253 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:33.921336 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:33.957159 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:34.079310 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 02:07:34.089972 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:34.093590 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:34.094992 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:34.095662 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:34.102532 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:34.139390 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:34.199867 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:34.216457 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:34.354310 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:34.364092 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:34.388818 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:07:34.389693 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 02:07:34.400668 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:34.400811 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:34.401619 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:34.402868 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:34.403740 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:34.422080 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:34.453399 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:34.496694 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:34.679141 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 02:07:34.709254 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:34.710072 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:34.711987 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:34.721184 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:34.725124 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:34.876243 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:34.936208 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:34.947819 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:34.955654 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:34.978137 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:35.011785 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:35.013689 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:35.014475 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:35.015847 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:35.027076 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:35.031050 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:35.135157 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:35.198338 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:35.256116 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:35.307353 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 02:07:35.320422 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:35.321077 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:35.323576 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:35.326069 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:35.328150 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:35.498487 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:35.538621 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:35.545811 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:35.552695 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:35.625887 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:35.628157 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:35.628612 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:35.631297 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:35.632377 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:35.659359 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:35.708122 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 02:07:35.919860 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 02:07:35.921350 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:35.933966 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:35.936911 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:35.938380 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:35.938553 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:35.942708 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:35.956133 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:36.014571 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:36.122773 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:36.245520 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:36.245846 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:36.249765 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:36.250561 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:36.257333 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 02:07:36.266542 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:36.430006 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:36.471757 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:36.498741 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:36.515604 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:36.551908 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 02:07:36.562450 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:36.565690 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 02:07:36.566493 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:36.568244 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:36.568641 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:36.570536 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:36.620389 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:36.627705 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:36.659622 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:07:36.660291 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:07:36.798753 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:07:36.857370 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:36.875224 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:36.877049 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:36.877118 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:36.879067 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:36.886837 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:37.018007 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:37.028757 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:37.033005 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:37.110486 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:37.181576 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:37.185046 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:37.191033 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 02:07:37.197327 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:37.198225 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:37.215764 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:37.268822 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:37.503213 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:37.506065 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:37.510814 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:37.511592 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:37.513636 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:37.539742 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:37.540092 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:37.569365 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:37.600857 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:37.675449 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:37.720741 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:37.821519 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:37.822010 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 02:07:37.824890 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:37.824898 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:37.825026 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:37.826668 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:37.827138 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:37.862498 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:37.874698 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:38.002718 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 02:07:38.030416 20489 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:07:38.030582 20489 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:07:38.031051 20489 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:07:38.032174 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:38.032315 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:38.032397 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:38.032531 20489 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:07:38.098547 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:07:38.099339 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:38.118227 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:38.139194 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:38.140106 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:38.142556 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:38.145066 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:38.151482 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:38.267035 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 02:07:38.434019 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 02:07:38.448586 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:38.449928 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:38.452241 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:38.453094 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:38.455598 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:38.489964 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:38.697626 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:07:38.704300 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:38.719089 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:38.758252 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:38.760553 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:38.760603 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:38.760983 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:38.765748 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:38.902942 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:07:38.992753 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:39.066409 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:39.067826 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:39.069248 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:39.070490 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:39.071465 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:39.146232 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:07:39.161145 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:07:39.190724 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:07:39.246129 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:39.291167 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:07:39.331707 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:39.374048 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:39.377314 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:07:39.379827 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:39.380203 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:39.383504 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:39.399382 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:39.463677 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:39.527073 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:07:39.527789 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:07:39.625370 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:07:39.643968 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 02:07:39.692903 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:39.694196 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:39.695626 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:39.702044 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:39.702607 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:39.758340 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:39.785400 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:39.792573 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 02:07:39.997643 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:40.001013 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:40.003563 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:40.004796 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:40.015169 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 02:07:40.015384 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:40.016233 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:40.167809 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:07:40.168460 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:07:40.205080 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:07:40.245402 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 02:07:40.298842 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:40.304183 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:40.306890 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:40.307624 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:40.307731 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:40.311093 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:40.320083 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:07:40.353893 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:40.454954 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:40.489849 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 02:07:40.615765 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:40.617408 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:40.617934 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:40.627332 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:40.629551 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:40.711772 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:40.838073 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:40.915613 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:07:40.924146 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:40.924834 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:40.925209 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:40.927340 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:40.929469 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:40.934608 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:40.975430 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:07:40.984505 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:07:41.148384 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:41.148746 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 02:07:41.234284 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:41.235838 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:41.237694 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:41.241060 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:41.241420 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:41.293287 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:07:41.390688 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:41.420800 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:41.540051 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:07:41.547294 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:41.548377 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:41.549729 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:41.553653 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:41.554579 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:41.623512 20883 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:07:41.623642 20883 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:07:41.624118 20883 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:07:41.625098 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:41.625181 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:41.625249 6849 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:41.625375 20883 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:07:41.638370 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:41.648192 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:41.708236 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 02:07:41.708937 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 02:07:41.709487 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:07:41.725162 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:07:41.807921 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 02:07:41.854696 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:41.857518 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:41.862566 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:41.869807 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:41.872727 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:41.894080 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:41.968194 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:42.047981 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:07:42.165475 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:42.168960 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:42.168972 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:42.169586 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:42.189795 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:42.191931 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:07:42.192684 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:42.248522 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:07:42.311079 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:07:42.478175 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:42.483175 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:42.486531 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:42.486567 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:42.495170 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:42.506948 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:42.508611 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:42.510568 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:07:42.523666 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:07:42.589005 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:07:42.625638 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:42.660615 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:07:42.750598 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:07:42.791818 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 02:07:42.794437 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:42.796027 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:42.798331 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:42.804811 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:42.807057 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:42.894107 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:43.024125 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:43.063196 20622 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:07:43.063354 20622 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:07:43.063843 20622 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:07:43.064939 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:43.065984 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:43.066048 8445 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:43.066179 20622 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:07:43.098678 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:43.098995 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:43.101589 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:43.102775 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:43.103019 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:43.105005 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:43.191541 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:43.248392 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:43.405117 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 02:07:43.408317 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:43.408757 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:43.409121 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:43.409796 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:43.411692 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:43.472224 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:43.561554 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:43.676177 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:43.711802 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:43.716678 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:43.717366 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:43.719100 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:43.723724 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:43.730728 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 02:07:43.730743 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:43.800133 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:43.868481 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:43.905489 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:43.931696 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 02:07:44.018520 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:44.019716 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:44.021229 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:44.027300 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 02:07:44.029735 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:44.030411 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:44.080909 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:44.150154 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:44.191474 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:44.285234 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:44.323231 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:44.323705 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:44.324955 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:44.326702 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:44.328400 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:44.364351 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:44.590219 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:44.590700 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 02:07:44.625471 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:44.628822 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:44.628918 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:44.639031 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:44.639110 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:44.646260 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:44.684779 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:44.728549 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:44.764868 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:44.800601 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:44.836125 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:44.898695 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 02:07:44.931222 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:44.932260 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:44.934478 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:44.935251 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:44.945459 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:44.955744 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:44.969159 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:45.091441 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:45.138314 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:45.181154 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:45.204326 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:45.243427 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:45.244901 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:45.245728 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:45.247247 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:45.249806 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:45.307222 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:45.384013 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:45.384896 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:45.455366 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 02:07:45.544317 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:45.546291 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:45.546952 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:45.546972 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:45.566994 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:45.598879 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:45.672677 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:45.699940 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:45.850289 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:45.851891 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:45.852279 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:45.864537 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:45.865576 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:45.991714 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 02:07:46.011549 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:46.016242 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:46.158021 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:46.160194 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:46.162448 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:46.169302 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:46.170006 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:46.172386 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:46.179986 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:46.208511 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:46.278266 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 02:07:46.368872 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 02:07:46.465324 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:46.465420 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:46.466573 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:46.468768 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:46.469250 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:46.499840 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:46.559049 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:46.580157 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:46.631896 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:46.676112 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:46.693599 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 02:07:46.766872 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:46.768529 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:46.769928 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:46.770380 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:46.779052 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:46.792778 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:07:46.884332 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:46.904654 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 02:07:46.921005 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:46.922127 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 02:07:47.073248 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:47.073609 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:47.074362 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:47.074542 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:47.076583 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:47.083810 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:47.138736 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:47.144080 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:47.368599 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 02:07:47.381093 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:47.382328 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:47.383280 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:47.384940 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:47.393607 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:47.403587 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:47.450011 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:47.450009 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:47.469820 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:47.508318 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:47.623836 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:47.693068 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:47.693181 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:47.693816 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:47.696795 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:47.698069 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:47.706549 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:47.707986 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:47.909976 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:47.953326 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:07:47.956019 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:47.997587 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 02:07:47.999300 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:47.999369 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:48.001355 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:48.012673 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:48.013819 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:48.058578 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:07:48.088107 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:48.221319 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:48.303474 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:48.304440 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:48.304786 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:48.307300 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:48.309566 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:48.350122 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:48.457186 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:07:48.522326 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:07:48.599527 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:48.613485 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:48.613911 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:48.614845 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:48.614935 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:48.625399 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:48.740710 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:07:48.767194 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:48.862921 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 02:07:48.929675 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:48.930128 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:48.930557 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:48.930579 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:48.937367 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:48.937995 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:49.019752 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 02:07:49.098757 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:49.107959 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:49.187634 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:07:49.208633 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:49.224634 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 02:07:49.236472 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:49.239817 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:49.240315 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:49.252405 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:49.254285 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:49.347149 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:07:49.366852 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:07:49.406231 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:49.481775 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:49.494467 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 02:07:49.547662 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:49.551996 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:49.552268 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:49.552515 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:49.552829 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:49.567409 6627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 02:07:49.578769 6627 catalog_manager.cc:4573] TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dcc5daef41bc44c4855f2bcb44e1c6d5: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:07:49.599334 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:49.613435 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:07:49.718427 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:49.743583 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:07:49.832458 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 02:07:49.859336 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:49.859534 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:49.870776 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:49.872107 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:49.875183 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:49.985757 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:49.990933 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:50.136979 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:50.137149 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:07:50.166424 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 02:07:50.169190 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:50.171283 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:50.173966 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:50.174679 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:50.184423 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:50.185185 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:50.343951 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:07:50.416546 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:50.432408 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:50.485849 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:50.486871 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:50.487645 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:50.488407 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:50.490586 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:50.536062 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:07:50.581784 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:07:50.622483 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 02:07:50.791812 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:50.792640 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:50.796497 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:50.796959 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:50.811348 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:50.814299 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:50.829993 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:50.834375 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:50.964807 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:51.100833 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:51.101799 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:51.102418 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:51.102555 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:51.106312 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:51.123961 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:07:51.143916 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:51.230417 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:51.302027 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:51.361903 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 02:07:51.421365 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:51.426669 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:51.431409 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:51.433663 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:51.434922 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:51.454048 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:51.504443 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:07:51.581730 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:51.627839 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 02:07:51.657861 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 02:07:51.733904 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:51.735693 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:51.737416 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:51.737565 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:51.745985 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:51.748929 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:51.792724 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:51.832110 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:51.837594 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:51.896088 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 02:07:51.922945 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:51.943248 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:52.042382 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:52.043900 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:52.044358 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:52.047740 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:52.053786 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:52.078907 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 02:07:52.141117 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:52.141314 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:52.155194 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:52.294220 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:52.362610 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:52.362979 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:52.364588 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:52.364749 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:52.375180 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:52.407123 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:52.422765 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:52.467252 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:07:52.526665 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:52.655798 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:52.672840 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:52.672922 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:52.674726 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:52.684547 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 02:07:52.685937 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:52.686549 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:52.697032 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:52.720019 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:52.807448 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:52.819847 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 02:07:52.836437 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 02:07:52.980325 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:52.981541 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:52.981856 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:52.985190 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:52.993906 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:52.994235 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 02:07:53.085016 22373 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:07:53.085160 22373 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:07:53.085652 22373 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:07:53.086736 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:53.087167 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:53.087239 7117 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:53.087467 22373 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:07:53.109046 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:53.213270 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 02:07:53.285435 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:53.287885 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:53.288565 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:53.288864 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:53.295697 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:53.297108 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:53.337631 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:53.587183 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:53.593612 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:53.593993 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:53.594761 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:53.595250 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:53.608850 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:53.713219 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:07:53.742795 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:07:53.743443 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:07:53.843441 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:53.885563 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:53.902531 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:53.903003 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:53.903177 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:53.906932 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:53.907542 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:53.962827 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:07:53.998625 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:07:54.053300 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:54.177093 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:07:54.185495 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:07:54.202327 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 02:07:54.213531 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:54.213913 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:54.216862 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:54.218607 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:54.221709 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:54.297478 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:07:54.360738 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:54.382134 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:07:54.392567 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:54.461964 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:07:54.501008 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 02:07:54.516984 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:54.517352 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:54.521150 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:54.521447 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:54.528918 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:54.537609 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:54.730525 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240502 02:07:54.827286 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:54.828382 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:54.829375 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:54.829455 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:54.838183 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:54.905026 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:54.923342 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:07:54.983592 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:07:55.003676 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:55.014320 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:55.040720 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:07:55.133651 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:07:55.137511 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 02:07:55.141310 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:55.142351 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:55.145020 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:55.145417 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 02:07:55.152324 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:55.165905 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:55.361152 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:07:55.416605 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:55.437320 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:55.457278 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:55.462841 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:55.464951 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:55.466358 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:55.467793 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:07:55.498663 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:55.499305 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:07:55.523003 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 02:07:55.772657 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:55.773736 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:55.774659 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:55.775175 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:55.775405 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:55.891978 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:55.919282 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:55.976415 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:07:56.027069 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:56.079568 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:56.082589 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:56.083498 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:56.083657 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:56.084756 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:56.192902 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:07:56.216418 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 02:07:56.226456 22326 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:07:56.226593 22326 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:07:56.227066 22326 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:07:56.228189 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:56.228296 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:56.228391 8445 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:56.228526 22326 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 02:07:56.231420 22677 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:07:56.231528 22677 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:07:56.231930 22677 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:07:56.232921 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:56.233021 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:56.233101 7117 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:56.233233 22677 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:07:56.261305 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:07:56.337483 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:56.377496 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:56.381215 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 02:07:56.392879 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:56.397492 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:56.401973 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:56.402634 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:56.411890 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:56.501988 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:56.584159 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:07:56.604303 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:07:56.611539 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 02:07:56.629145 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 02:07:56.709425 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:56.709930 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:56.710577 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:56.712493 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:56.726387 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:56.752593 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:56.775898 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:07:56.855264 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:07:56.909483 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 02:07:56.910158 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:56.944133 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:56.948033 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 02:07:57.019032 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:57.019182 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:57.020108 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:57.021075 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:57.021688 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:57.324019 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:57.324174 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:57.324836 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:57.325317 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:57.325902 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:57.336225 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:57.356648 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:57.363462 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:57.411866 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:07:57.476326 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:07:57.505410 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 02:07:57.593160 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 02:07:57.639266 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:57.639925 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:57.640031 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:57.641417 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:57.644371 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:57.693460 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:07:57.713557 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:07:57.852622 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:57.892405 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:07:57.910151 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:57.929850 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 02:07:57.947120 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:57.950484 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:57.951107 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:57.953244 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:57.967247 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:57.982216 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:07:57.985456 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:58.085322 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:07:58.242671 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:58.255468 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:58.258596 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:58.259095 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:58.259316 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:58.262313 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:58.290721 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:07:58.332330 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:58.569288 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:58.570040 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:58.573096 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:58.573427 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:58.574980 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:07:58.601938 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:07:58.608000 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:58.661860 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:07:58.762665 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:58.841790 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:07:58.842878 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:58.879480 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:58.880021 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:58.882182 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:58.882759 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:58.883301 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:58.890296 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 02:07:58.987735 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:07:59.079480 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:59.103849 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:07:59.104851 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:07:59.185017 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:59.188007 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:07:59.188205 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:59.195940 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:59.196173 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:59.206604 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:59.298475 22923 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:07:59.298626 22923 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:07:59.299001 22923 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:07:59.300052 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:59.300135 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:07:59.300184 6849 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:07:59.300316 22923 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:07:59.307271 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:07:59.368139 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:07:59.419991 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:07:59.465029 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:07:59.494771 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:59.499054 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:59.502274 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:07:59.502542 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:59.510468 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:07:59.534904 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:07:59.566932 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:07:59.577592 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:59.769892 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 02:07:59.808770 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:07:59.810204 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:07:59.813763 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:07:59.814491 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:07:59.815127 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:07:59.857048 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:07:59.987592 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:07:59.990450 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:00.037730 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:00.041957 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 02:08:00.114969 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:00.116673 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:00.120160 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:00.120318 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:00.125929 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:00.130383 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:08:00.181205 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:08:00.186687 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:08:00.388351 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:00.406458 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:08:00.424327 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:00.427361 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:00.428599 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:00.438319 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:00.443009 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:00.512292 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:00.541510 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:00.563724 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:08:00.670552 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 02:08:00.731016 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:00.732836 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:00.735307 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:00.735877 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:00.735975 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:00.796833 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:08:00.939373 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:00.979701 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:00.984648 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:08:00.990550 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:01.039598 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:01.040201 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:01.043200 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:01.051658 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:01.059674 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:08:01.060429 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:01.107748 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:01.228703 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 02:08:01.353986 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:01.356765 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:01.356753 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:01.357161 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:01.358146 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:01.359398 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:01.406973 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:01.486469 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:08:01.511548 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:01.512641 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 02:08:01.564235 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:01.582548 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:08:01.614257 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:08:01.669682 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:01.673004 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:01.675506 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:01.676010 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:01.697894 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:01.791546 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:08:01.917637 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:08:01.918305 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:01.969130 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:01.986595 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:01.986824 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:01.987262 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:01.988238 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:01.988385 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:01.992069 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:02.004875 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:08:02.265286 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:08:02.295203 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:02.295966 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:02.296021 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:02.297384 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:02.298115 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:02.324735 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:02.412370 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:02.459172 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:02.498795 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:02.561025 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:08:02.604344 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:02.604851 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:02.606725 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:02.606803 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:02.607553 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:02.677075 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:08:02.721715 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:08:02.873545 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:02.908963 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:02.910365 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:02.912444 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:02.922610 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:02.922856 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:02.980726 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 02:08:02.981554 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:02.988503 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:03.007216 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:08:03.011632 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:03.068361 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:08:03.075925 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:03.150723 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:08:03.218989 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:03.221887 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:03.222874 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:03.222947 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:03.224864 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:03.431813 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:03.444352 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:03.451969 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:03.529726 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:03.533474 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 02:08:03.534368 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:03.536789 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:03.538890 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:03.551630 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:03.675004 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:03.737696 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:03.789245 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:03.807341 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:03.843003 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:03.845445 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:03.846490 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 02:08:03.846513 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:03.847677 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:03.848389 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:03.848903 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:03.849606 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:03.881153 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:03.891654 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:03.985273 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:04.015686 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:08:04.086141 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 02:08:04.158284 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:04.159582 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:04.159920 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:04.160848 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:04.174710 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:04.275812 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:04.292797 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:04.458261 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:04.472249 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:04.477167 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:04.480465 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:04.485464 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:04.485822 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:04.684299 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:04.706871 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:04.715167 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:04.782760 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:04.782784 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:04.787989 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:04.787988 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:04.793804 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 02:08:04.796113 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:04.818605 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:04.856017 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:04.898907 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:04.989019 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:05.019261 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:05.085561 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:05.088156 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:05.090849 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:05.091801 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:05.100955 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:05.184752 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:05.227216 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:05.248369 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:05.404286 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:05.407886 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:05.410187 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:05.421142 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:05.422228 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:05.497972 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:05.556521 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:05.565876 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:05.574621 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:05.614727 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:05.674849 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:05.685921 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:05.723523 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:05.723722 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:05.725586 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:05.726549 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:05.740387 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:05.763200 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:05.927438 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:08:06.039464 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:06.041370 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:06.044091 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:06.044245 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:06.047385 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:06.051759 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:06.087268 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:06.221721 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:06.297359 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:08:06.298440 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:06.359922 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:06.366477 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:06.367115 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:06.367383 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:06.368304 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:06.419582 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:08:06.440059 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:06.468941 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:06.517655 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:06.532914 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:06.565577 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:06.571213 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:08:06.670109 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:06.671527 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:06.674867 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:06.676148 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:06.678074 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:06.685724 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:06.702721 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:08:06.847730 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:06.917693 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:06.957396 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 02:08:06.982161 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:06.983741 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:06.984180 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:06.989192 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:06.996570 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:08:07.001672 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:07.073140 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:07.109407 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:07.171429 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:08:07.293257 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:07.294646 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:07.296970 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:07.304080 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:07.314677 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:07.332520 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:07.376622 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:08:07.420156 24425 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:08:07.420331 24425 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:08:07.420815 24425 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:08:07.421895 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:07.422142 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:07.422231 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:07.422369 24425 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:08:07.530611 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:07.537351 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:07.599498 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:07.601931 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:07.602118 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:07.607693 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:07.612030 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:07.626706 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:07.701097 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:07.807996 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:07.914062 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:08:07.915004 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:07.915241 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:07.916555 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:07.923065 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:07.928941 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:07.950896 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:08.060237 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:08.103902 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:08.121812 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:08.142683 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:08:08.224900 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:08.227823 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:08.231391 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:08.232432 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:08.238991 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:08.243829 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:08.361562 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:08:08.374719 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:08:08.386674 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:08.430821 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 02:08:08.445613 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:08:08.533005 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:08.533752 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:08.546213 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:08.546808 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:08.550730 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:08.655807 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:08.688748 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:08.838495 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:08.841259 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:08.842096 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:08.850934 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:08.869938 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:08.887969 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:08.946579 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:09.065317 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:08:09.093137 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:08:09.110189 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:08:09.158577 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:09.161089 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:09.161592 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:09.164783 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:09.166589 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:09.195480 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:08:09.218919 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:09.249953 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:08:09.250732 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:09.282979 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:09.319437 24766 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:08:09.319607 24766 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:08:09.320073 24766 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:08:09.321089 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:09.321183 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:09.321293 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:09.321442 24766 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:08:09.479763 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:09.479792 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:09.482003 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:09.486591 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:09.498437 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:08:09.499095 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:09.631404 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:09.659631 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:09.680902 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:08:09.736339 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:09.779572 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:09.787463 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:09.788728 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:09.790014 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:09.791087 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:09.796424 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:09.849335 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:08:09.876183 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:08:09.898818 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:08:09.957809 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:08:10.023758 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:08:10.090737 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:10.094239 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:10.095279 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:10.103648 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:10.103719 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:10.127601 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:08:10.231213 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:10.308668 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:10.325731 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:10.344404 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:10.399288 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:10.399917 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:10.401360 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:10.404583 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:10.404642 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:10.419922 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:08:10.537353 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:08:10.707676 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:10.707916 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:10.709399 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:10.711273 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:10.711694 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:10.712411 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:10.738868 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:10.765547 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:10.770763 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:10.843962 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:10.908227 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:08:10.910774 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:08:11.013024 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:11.014935 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:11.014940 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:11.016302 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:11.028039 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:11.146889 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:11.278216 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:11.279487 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:11.303876 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:08:11.305148 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:08:11.321208 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:11.326877 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:11.331095 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:11.331511 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:11.331699 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:11.580724 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:11.639328 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:11.640610 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:11.642902 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:11.644372 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:08:11.646579 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:11.646852 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:11.674453 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:11.683347 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:11.683445 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:08:11.799887 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:11.830080 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:11.931033 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:08:11.962916 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:11.963037 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:11.963663 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:11.965170 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:11.984405 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:12.081573 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:08:12.082299 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:12.086551 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:08:12.109778 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:12.207273 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:12.234606 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:08:12.271605 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:12.272001 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:12.272053 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:12.283730 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:12.284142 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:12.307248 22923 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:08:12.307381 22923 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:08:12.307770 22923 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:08:12.308751 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:12.308849 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:12.308929 6849 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:12.309079 22923 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:08:12.342001 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:12.406359 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:12.444192 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:12.518222 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:12.556048 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:12.573215 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:12.574479 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:12.576396 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:12.583582 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:12.585011 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:12.655859 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:12.714720 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:12.734969 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:12.764873 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:12.800318 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:08:12.891080 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:12.894029 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:12.894512 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:12.894984 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:12.895337 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:12.944366 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:13.070849 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:13.207053 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:13.209039 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:13.211753 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:13.212139 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:13.212785 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:13.222316 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:08:13.263330 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:08:13.339942 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:13.418107 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:08:13.478472 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:08:13.514987 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:13.515075 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:13.518312 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:13.525226 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:13.539842 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:13.542979 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:13.601681 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:13.713418 25282 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:08:13.713580 25282 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:08:13.714056 25282 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:08:13.714448 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 02:08:13.715108 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:13.715198 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:13.715279 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:13.715427 25282 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:08:13.758285 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:13.835433 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:13.837566 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:13.842037 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:13.842944 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:13.844208 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:13.879947 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:13.986629 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:14.037788 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:14.075376 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:08:14.158623 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:14.159379 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:14.160053 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:14.170405 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:14.176262 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:14.179431 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:08:14.197396 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:08:14.213914 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:08:14.269165 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:08:14.269867 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:14.437871 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:14.454990 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:08:14.460837 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:14.475343 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:14.477444 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:14.477983 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:14.479825 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:14.484779 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:14.640540 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:14.767839 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:08:14.781778 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:14.782891 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:14.783397 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:14.792812 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:14.793398 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:14.827342 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:14.842696 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:14.852507 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:08:14.867116 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:08:14.891463 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:14.975689 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:08:14.997277 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:08:15.080440 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:15.083668 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:08:15.086189 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:08:15.095391 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:15.095605 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:15.096691 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:15.097421 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:15.109014 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:15.167598 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:15.249799 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:08:15.371838 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:15.405227 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:15.406452 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:15.411582 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:15.412597 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:15.420250 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:15.469727 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:15.558384 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:08:15.677237 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:15.718189 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:15.722738 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:15.724805 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:15.736661 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:15.737427 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:15.771318 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:15.850011 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:16.031340 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:16.033010 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:16.037428 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:16.038693 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:16.039225 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:16.083056 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:16.163393 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:08:16.171515 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:16.234570 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:16.237635 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:16.272779 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:16.328262 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 02:08:16.343663 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:16.347339 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:16.347429 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:16.347474 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:16.347828 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:16.417310 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:16.606026 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:16.643272 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:16.651111 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:08:16.652891 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:16.653291 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:16.655002 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:16.655709 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:16.659300 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:16.678381 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:16.688644 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:16.807163 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:16.932741 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:08:16.965871 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:16.974067 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:16.974637 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:16.977181 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:16.990471 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:16.991024 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:17.081537 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:17.091781 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:08:17.092489 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:17.282971 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:17.286525 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:17.287144 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:17.288547 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:17.297609 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:17.333110 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:08:17.335484 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:17.370677 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:17.487808 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:17.505415 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:17.533092 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:17.551267 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:17.558725 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 02:08:17.593392 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:17.595661 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:17.597831 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:17.599380 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:17.603724 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:17.608247 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:17.626004 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:17.684437 26111 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
W20240502 02:08:17.684543 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 02:08:17.684595 26111 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:08:17.685034 26111 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:08:17.686161 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:17.686446 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:17.686527 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:17.686682 26111 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:08:17.717517 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:08:17.877207 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:17.916941 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:17.917233 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:17.918251 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:17.919075 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:17.920162 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:18.075615 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:18.079430 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:08:18.183327 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:18.227542 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:18.230661 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:18.231786 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:18.236475 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:18.249014 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:18.323725 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:18.348850 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:08:18.545634 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:18.548806 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:18.550355 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:18.550511 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:18.560194 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:18.565999 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:18.644668 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:08:18.757694 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:18.849750 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:18.853484 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:08:18.875479 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:18.877214 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:18.878820 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:18.881290 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:18.881649 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:18.972460 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:08:19.086926 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20240502 02:08:19.164198 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:19.171675 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 02:08:19.187635 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:19.190646 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:19.190972 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:19.193282 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:08:19.205251 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:19.206864 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:19.215342 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:08:19.260191 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:08:19.276573 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:19.366762 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:08:19.387984 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:19.428159 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240502 02:08:19.506609 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:19.511032 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:19.512512 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:08:19.520016 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:19.520749 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:19.524142 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:19.652894 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:08:19.711720 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:19.783846 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:19.827344 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:08:19.829658 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:19.831097 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:19.831110 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:19.835935 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:19.839299 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:19.869395 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:08:19.869411 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:08:19.919194 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:08:19.976934 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:08:19.978821 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:20.083372 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:08:20.138238 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:08:20.152917 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:20.152917 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:20.155433 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:20.155436 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:20.155514 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:20.234310 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:08:20.304374 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:20.331856 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:20.367228 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:20.384637 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:08:20.429292 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:08:20.465909 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:20.467558 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:20.467658 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:20.467869 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:20.471241 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:20.780431 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:20.780987 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:20.782465 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:20.783303 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:20.784458 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:20.816092 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:20.840857 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:08:20.881074 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:20.935930 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:21.085294 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:21.088765 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:21.089900 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:21.090314 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:21.098603 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:21.243467 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:21.275153 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:21.339926 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:08:21.381570 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:21.399729 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:21.400120 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:21.400840 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:21.401504 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:21.402056 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:21.404031 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:21.441196 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:08:21.464553 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:08:21.702327 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:21.705597 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:21.708173 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:21.717011 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:21.719475 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:21.742836 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:08:21.769845 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:08:21.835562 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:08:21.835757 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:21.857226 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:21.920066 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:21.939028 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:08:22.004331 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:22.018158 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:22.021029 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:22.022198 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:22.025616 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:22.029496 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:22.045221 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:08:22.090668 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:08:22.103003 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 02:08:22.148689 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:22.316800 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:08:22.329941 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:22.330281 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:22.333698 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:22.335177 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:22.335469 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:22.390677 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:22.450960 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:22.505345 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:08:22.519095 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:08:22.535941 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:22.571722 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:08:22.583372 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:08:22.593165 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:08:22.636623 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:22.639242 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:22.643419 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:22.643461 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:22.656015 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:22.685221 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:22.845556 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:22.879319 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:22.926302 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:22.954506 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:22.955987 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:22.956614 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:22.957583 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:22.958448 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:23.091945 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:08:23.266368 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:23.266476 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:23.268270 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:23.269515 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:23.288136 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:23.364742 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:23.404824 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:23.459815 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:23.465131 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:23.574707 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:23.575001 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:23.578580 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:23.578992 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:23.586676 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:08:23.588027 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:23.795922 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:23.821067 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:08:23.884613 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:23.885107 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:23.886838 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:23.887378 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:23.894572 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:23.904970 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:23.915045 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:23.920410 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:23.928702 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:08:24.201704 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:24.203194 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:24.203583 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:24.205390 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:24.216069 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:24.225018 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 02:08:24.244421 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:24.280663 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:24.302085 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:24.338191 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:24.359167 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:24.511098 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:24.514891 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:24.515530 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:24.526916 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:24.539219 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:08:24.539846 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:24.548633 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:24.594421 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:24.672772 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:24.722020 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:24.732731 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:24.784879 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:24.785130 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:24.821980 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:24.826537 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:08:24.828006 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:24.829949 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:24.830155 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:24.833606 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:24.843832 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:24.951290 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:25.028605 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:25.033684 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:08:25.142190 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:25.143522 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:25.146256 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:25.147924 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:25.148337 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:25.155169 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:25.206647 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:08:25.224386 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:25.245008 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:25.336478 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:25.351210 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:25.353137 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:08:25.453043 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:25.453882 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:25.454080 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:25.458287 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:25.464543 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:25.676488 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:08:25.706269 27072 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:08:25.706434 27072 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:08:25.706914 27072 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:08:25.707856 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:25.708093 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:25.708173 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:25.708310 27072 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:08:25.761654 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:25.761848 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:25.765228 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:25.770571 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:25.771672 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:25.792205 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:25.852778 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:25.927789 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:26.068172 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:26.070022 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:26.071175 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:26.071985 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:26.072682 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:26.186857 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:26.208794 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:26.254766 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:26.341681 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:26.375020 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:26.380367 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:26.382561 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:26.384892 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:26.387022 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:26.391252 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:26.444545 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:26.453665 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:08:26.466302 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:08:26.535997 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:08:26.695598 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:26.697734 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:26.699514 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:26.702562 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:26.710834 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:26.733480 27256 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:08:26.733657 27256 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:08:26.734108 27256 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:08:26.735415 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:26.736234 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:26.736299 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:26.736478 27256 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:08:26.771607 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:26.797170 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:26.812072 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:26.873226 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:26.953624 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:27.002203 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:27.002947 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:27.008284 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:27.008690 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:27.015504 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:27.112051 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:27.112061 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:27.201977 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:27.216542 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:27.220623 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:08:27.221279 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:27.304692 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:08:27.314364 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:27.315184 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:27.316256 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:27.319614 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:27.322603 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:08:27.329659 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:27.354902 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:08:27.359289 27451 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:08:27.359431 27451 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:08:27.359846 27451 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:08:27.360913 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:27.361027 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:27.361080 7117 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:27.361217 27451 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:08:27.395277 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:27.417843 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:27.466833 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:27.562662 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:27.576082 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:08:27.623925 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:27.625789 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:27.626693 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:27.628378 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:27.628934 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:27.677001 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:27.709734 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:27.812702 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:27.846472 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:27.936571 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:27.937973 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:27.940907 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:27.941655 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:27.947386 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:27.982188 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:28.153930 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:28.167919 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:08:28.250478 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:28.250752 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:28.250948 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:28.253536 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:28.256827 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:28.281380 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:28.364305 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:28.434747 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:28.481163 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:28.545500 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:28.559571 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:28.561434 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:28.564616 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:28.564757 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:28.572252 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:28.763470 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:28.818683 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:28.832149 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:08:28.883875 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:28.886159 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:28.887887 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:28.888015 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:28.889351 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:29.035882 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:08:29.151393 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:29.182760 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:08:29.195932 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:29.199143 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:29.200342 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:29.200790 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:29.203519 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:29.295059 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:29.343899 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:29.347419 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:29.467753 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 02:08:29.512728 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:29.513160 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:29.515082 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:29.515652 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:29.526646 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:29.527873 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:29.656173 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:29.674844 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:08:29.675609 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:29.705899 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:29.780424 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:29.791965 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:29.800568 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:29.829807 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:29.831631 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:29.833614 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:29.840912 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:29.844034 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:29.845990 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:29.856424 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:29.933267 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:29.938759 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:29.967180 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:29.979838 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:08:30.152824 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:30.153605 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:30.154438 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:30.154881 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:30.155176 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:30.165074 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:30.307377 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:08:30.308068 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:30.340118 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:30.392895 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:30.407867 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:30.483302 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:30.486127 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:30.487877 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:30.496448 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:30.501188 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:30.578665 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:30.706460 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:30.717113 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:30.797132 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:30.797451 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:30.798763 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:30.801015 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:30.801159 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:30.942489 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:30.943480 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:30.956347 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:31.076898 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:08:31.108327 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:31.108829 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:31.110868 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:31.122717 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:31.124631 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:31.263337 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:31.333230 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:31.402794 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:31.432647 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:31.433235 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:31.435441 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:31.435748 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:31.437528 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:31.460861 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:08:31.514439 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 02:08:31.546255 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:31.751382 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:31.754664 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:31.754686 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:31.755489 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:31.759416 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:31.792155 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:31.792858 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:31.863915 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:31.866568 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:31.902967 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:31.961565 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:32.063062 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:32.065841 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:08:32.074296 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:32.075718 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:32.076267 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:32.076817 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:32.288347 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:32.342435 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:08:32.343039 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:32.360265 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:32.365558 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:32.365833 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:32.377650 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:08:32.379444 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:32.379719 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:32.381408 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:32.382465 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:32.383355 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:32.473313 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:32.516865 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:32.524821 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:32.584128 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:08:32.690590 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:32.694804 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:32.698648 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:32.703641 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:32.713826 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:32.721894 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:32.779896 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:32.831332 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:32.837399 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:32.964799 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:33.010574 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:33.011865 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:33.012985 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:33.020502 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:33.035007 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:33.140275 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:33.202808 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:33.229326 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:33.274183 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:33.324663 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:33.326218 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:33.326517 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:33.335065 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:33.335117 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:33.416301 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:33.420653 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:33.514163 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:08:33.628250 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:33.633527 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:33.637533 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:08:33.641010 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:33.646644 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:33.649989 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:33.677695 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:33.715716 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:08:33.760129 27735 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:08:33.760264 27735 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:08:33.760753 27735 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:08:33.761783 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:33.761883 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:33.761974 8445 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:33.762109 27735 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:08:33.828316 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:33.930822 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:33.955202 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:33.956148 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:33.957978 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:33.958773 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:33.958863 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:34.032652 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:08:34.098217 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:34.258523 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:34.259109 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:34.261118 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:34.272248 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:34.282932 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:34.297020 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:34.309116 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:34.334182 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:34.361884 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:34.463040 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:34.497514 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 02:08:34.523963 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:08:34.571560 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:34.572858 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:34.573238 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:34.575346 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:34.582600 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:34.664263 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:34.741665 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:34.742691 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:08:34.743270 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:34.774434 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:34.820861 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:08:34.880065 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:34.881258 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:08:34.882833 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:34.887622 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:34.890952 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:34.891356 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:34.951448 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:35.033097 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:35.047744 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:35.058429 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:35.110967 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:35.126312 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:35.129453 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:35.181623 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:35.191939 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:35.193493 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:35.193593 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:35.195854 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:35.197602 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:35.373905 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:35.468626 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 02:08:35.500023 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:35.501489 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:35.502501 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:35.505878 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:35.506708 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:35.589336 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:35.619868 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:35.651849 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:35.680406 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:35.756346 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:35.802307 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:35.807400 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:35.807796 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:35.808048 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:35.809598 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:35.876201 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:35.923934 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:08:36.110636 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:36.110720 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:36.110877 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:36.112416 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:36.126835 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:36.147622 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:36.208125 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:36.288049 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:36.297523 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:36.418326 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:36.419308 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:36.419613 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:36.422607 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:36.423300 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:36.432479 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:36.459547 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:08:36.578588 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240502 02:08:36.636780 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:36.711894 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:36.724918 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:36.733315 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:36.734555 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:36.736989 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:36.738060 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:36.755797 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:36.851663 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:36.960301 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:36.992827 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:37.029374 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:37.040879 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:37.041707 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:37.043350 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:37.044803 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:37.046129 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:37.054435 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:37.135999 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:37.213299 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:37.224449 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:37.291249 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:37.343627 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:37.344386 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:37.345067 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:37.347574 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:37.355259 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:37.365240 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:37.372443 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:37.496927 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 02:08:37.513056 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:37.526805 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:37.546725 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:37.593370 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:37.641455 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:37.658249 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:37.660611 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:37.660708 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:37.662499 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:37.674922 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:37.678854 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:37.692109 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:37.823882 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:37.939607 28519 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:08:37.939760 28519 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:08:37.940178 28519 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:08:37.941174 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:37.941256 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:37.941354 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:37.941612 28519 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:08:37.979941 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:37.982111 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:37.982141 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:37.983206 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:37.983209 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:38.078788 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:38.151300 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:38.267452 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:38.289664 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:38.292094 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:38.292721 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:38.295534 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:38.295868 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:08:38.300537 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:38.362443 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:38.381345 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:38.424264 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:38.536664 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:38.592151 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:38.593823 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:38.598160 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:38.599632 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:38.602581 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:38.694142 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:38.712602 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:08:38.904100 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:38.905661 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:38.908104 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:38.912200 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:38.923015 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:38.924533 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:38.947465 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:38.997042 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:08:39.015331 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:39.220788 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:39.221913 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:39.222477 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:39.227048 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:39.232288 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:39.236230 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:39.274101 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:39.337889 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:39.458724 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:08:39.465272 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:39.499374 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:39.527747 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:39.531935 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:39.538774 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:39.542948 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:39.544240 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:39.552454 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:39.558920 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:08:39.700251 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:39.821909 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:08:39.833261 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:39.845654 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:39.848134 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:39.848224 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:39.848484 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:39.849730 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:39.903250 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:08:39.903733 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:08:39.989971 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:39.991377 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:40.005345 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:40.024519 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:40.028549 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:40.034888 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:40.055770 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 02:08:40.136412 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:08:40.155932 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:40.158221 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:40.161340 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:40.163594 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:40.170576 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:40.197041 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:40.355840 29206 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:08:40.355996 29206 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:08:40.356520 29206 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:08:40.357656 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:40.357753 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:40.357837 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:40.357982 29206 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:08:40.412866 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:08:40.413230 28431 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:08:40.413350 28431 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:08:40.413754 28431 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:08:40.414563 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:40.414999 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:40.415102 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:40.415242 28431 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:08:40.420607 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:40.471151 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:40.472502 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:40.472612 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:40.477900 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:40.486685 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:40.525461 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:08:40.537063 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:40.643553 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:40.737902 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:08:40.779100 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:40.780248 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:40.780903 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:40.781950 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:40.782467 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:40.811686 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:40.825703 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:40.931299 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:40.993435 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:41.046099 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:41.083820 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:41.086766 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:41.088562 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:41.091331 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:41.095584 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:41.241266 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:08:41.300472 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:41.390803 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:41.394297 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:41.394413 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:41.394945 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:41.394963 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:41.497802 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:41.591305 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:08:41.593108 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:08:41.633615 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:41.696638 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:41.697264 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:41.699961 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:08:41.701107 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:41.706686 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:41.718786 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:41.726058 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:08:41.766196 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:41.881815 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:08:42.009508 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:42.010409 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:42.011250 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:42.018107 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:42.019433 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:42.028658 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:42.240191 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:42.245882 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:42.281644 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:08:42.314679 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:42.316278 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:42.317324 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:42.318014 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:42.327309 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:42.353345 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:42.356329 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:42.404176 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:08:42.409391 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:42.439968 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:08:42.451269 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:08:42.475498 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:08:42.489082 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:42.575446 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:42.585009 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:08:42.587131 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 77 similar messages]
I20240502 02:08:42.618599 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:42.619720 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:42.623443 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:42.627485 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:42.630978 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:42.635742 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:42.705116 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:08:42.812043 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:42.849197 29204 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:08:42.849350 29204 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:08:42.849762 29204 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:08:42.850787 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:42.850984 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:42.851078 6849 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:42.851230 29204 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:08:42.923722 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:42.924593 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:42.926771 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:42.935509 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:42.936863 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:42.989650 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:43.057384 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:08:43.074483 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:43.128419 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:43.232944 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:43.233696 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:43.237314 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:43.244690 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:43.246665 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:43.279606 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:43.310475 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:08:43.432988 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:08:43.536453 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:43.539420 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:43.540056 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:43.547688 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:43.551448 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:43.561496 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:43.591342 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:43.717180 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:43.846262 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:43.849368 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:43.849432 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:43.849805 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:43.850939 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:43.856618 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:43.956084 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:44.074999 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:08:44.158592 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:44.160755 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:44.163794 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:44.163988 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:44.164150 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:44.197940 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:44.219038 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:44.233589 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:44.434511 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:08:44.474072 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:44.476681 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:44.477763 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:44.479599 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:44.487370 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:44.552724 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:44.563812 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:44.612304 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:44.618453 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:08:44.784159 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:44.784459 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:44.787739 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:44.791095 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:44.799341 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:44.822674 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:44.943775 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:44.954648 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:44.955451 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:44.969617 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:44.999732 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:45.022017 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:45.024300 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:45.090914 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 02:08:45.092429 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:45.093394 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:45.093493 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:45.096761 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:45.099119 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:45.171418 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:45.173945 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:45.186290 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:45.320998 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:45.357905 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:45.399343 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:08:45.401083 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:45.405583 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:45.406042 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:45.408224 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:45.414755 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:45.532027 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:08:45.584830 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:45.640264 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:45.644495 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:45.646605 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 02:08:45.674765 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:45.676965 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:08:45.714556 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:45.721032 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:45.721522 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:45.721808 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:45.722613 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:45.828701 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:45.946875 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:08:46.026427 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:46.026944 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:46.027379 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:46.029104 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:46.043447 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:46.090708 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:46.129123 29206 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:08:46.129307 29206 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:08:46.129777 29206 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:08:46.130890 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:46.131068 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:46.131140 7117 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:46.131290 29206 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:08:46.204532 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:46.226522 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:46.333907 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:46.335464 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:46.340046 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:46.351445 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:46.351989 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:46.364913 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:46.512982 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:46.556859 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:08:46.620143 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:08:46.621708 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:46.645805 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:46.645936 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:46.647578 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:46.650270 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:46.656994 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:46.675273 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:46.713081 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:46.877967 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:46.917973 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:46.932580 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:08:46.952241 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:46.954222 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:46.955287 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:46.965745 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:46.975800 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:47.094511 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:47.132936 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:47.221999 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:47.264695 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:47.266328 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:47.266661 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:47.269204 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:47.270044 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:47.428601 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:47.483682 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:47.537499 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:08:47.572237 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:47.573051 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:47.578653 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:47.587168 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:47.587182 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:47.604739 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:47.630666 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:47.656644 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:08:47.662897 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:08:47.684345 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:08:47.684953 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:47.731560 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:47.824620 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:08:47.827059 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:47.862143 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:08:47.887953 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:47.888345 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:47.890190 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:47.890877 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:47.903259 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:48.026374 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:48.119899 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:48.137471 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:48.138934 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:48.156167 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:48.164078 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:48.203557 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:48.204107 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:48.204200 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:48.205864 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:48.206672 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:48.293684 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:48.363018 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:08:48.413945 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:48.510936 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:48.512377 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:48.518573 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:48.519443 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:48.525071 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:48.535251 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:08:48.553227 29776 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:08:48.553359 29776 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:08:48.553848 29776 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:08:48.554817 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:48.555335 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:48.555411 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:48.555557 29776 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:08:48.703342 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:48.725045 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:48.823835 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:48.825331 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:48.829907 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:48.831029 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:48.843941 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:49.031761 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:49.115968 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:08:49.119985 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:08:49.139581 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:49.141505 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:49.145449 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:49.149631 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:49.154997 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:49.156839 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:49.174963 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:49.218577 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:08:49.287240 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:08:49.444263 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:49.448168 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:49.450670 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:49.452497 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:49.476166 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:49.557096 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:49.584280 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:49.608356 6627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 02:08:49.609237 6627 catalog_manager.cc:4573] TS 4e6e76b381af4cdc90bd3e9e6e8b5cf2 (127.27.111.199:41741): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dcc5daef41bc44c4855f2bcb44e1c6d5: Network error: Client connection negotiation failed: client connection to 127.27.111.199:41741: connect: Connection refused (error 111)
W20240502 02:08:49.629788 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:49.644407 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:08:49.768445 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:49.771607 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:49.779562 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:49.779978 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:49.782740 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:49.809716 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:08:49.826092 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:08:49.945129 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:49.965070 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:50.039767 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:08:50.051656 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:50.076795 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:50.079871 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:08:50.079948 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:50.080626 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:50.082768 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:50.091161 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:50.220281 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:08:50.259922 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:08:50.272837 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 02:08:50.382444 29777 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:08:50.382596 29777 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:08:50.383085 29777 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:08:50.386231 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:50.386709 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:50.386801 6849 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:50.386947 29777 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:08:50.392374 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:50.393585 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:50.393803 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:50.395888 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:50.396839 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:08:50.404709 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:50.410547 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:50.473392 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:50.492990 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:08:50.520613 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:08:50.541739 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:50.611704 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 02:08:50.698442 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:50.702893 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:50.704053 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:50.705776 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:50.715198 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:08:50.720212 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:50.762213 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:08:50.770089 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:08:50.771682 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:08:50.800835 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:50.912850 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:50.926065 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:50.931682 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:08:50.947906 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:08:51.007481 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 02:08:51.008487 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:51.010033 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:51.011169 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:51.011744 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:51.015544 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:51.285852 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:51.316402 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:51.317704 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:51.318976 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:51.321069 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:51.335036 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:51.469882 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:08:51.473100 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:51.521342 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:51.571570 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:08:51.592140 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:08:51.629401 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:51.629565 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:51.632251 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:51.633294 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:51.633766 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:51.666698 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:51.718647 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:08:51.767902 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:51.917927 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:51.934664 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:51.935530 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:51.938119 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:51.941052 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:51.947794 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:52.046990 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:52.092622 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:52.240602 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:52.241995 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:52.244946 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:52.245471 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:52.246692 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:52.370491 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:52.391518 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:52.432905 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:08:52.436933 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:08:52.456611 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:08:52.524971 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:52.557870 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:52.558888 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:52.559787 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:52.560977 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:52.571362 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:52.575618 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:08:52.628626 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:52.675246 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:08:52.859624 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:52.860872 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:08:52.861927 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:52.862499 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:52.863096 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:52.863812 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:52.883551 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:52.936223 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 02:08:52.936803 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:08:52.974670 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:52.996439 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:53.143781 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:08:53.169128 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:53.170337 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:53.172436 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:53.174324 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:53.182251 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:53.236346 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:53.303642 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:08:53.315552 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:53.372047 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:08:53.393208 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:53.416785 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:08:53.483862 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:53.485090 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:53.487612 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:53.488750 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:53.491080 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:53.507673 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:53.613260 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:08:53.629750 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:08:53.786684 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:53.788187 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:53.789806 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:53.790251 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:53.794919 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:53.812180 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:53.847009 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:53.891444 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:53.982581 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:54.100273 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:54.103138 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:54.106928 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:54.107887 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:54.110618 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:54.170004 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:08:54.244302 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:54.276701 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:54.286073 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:54.296078 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:54.404947 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:54.414955 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:54.415624 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:54.415704 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:54.417572 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:54.417865 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:08:54.571420 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:54.669440 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:54.734329 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:54.735625 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:54.737017 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:54.737160 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:54.746285 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:54.803799 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:54.808753 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:54.922688 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:54.950206 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:55.039642 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:08:55.043704 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:55.043766 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:55.045969 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:55.048022 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:55.050875 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:55.072125 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:55.144126 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:55.182504 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:55.188639 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:55.251381 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:55.305143 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:55.346007 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240502 02:08:55.349753 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:55.353447 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:55.357151 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:55.357638 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:55.358033 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:55.359588 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:55.402532 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:55.669829 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:55.672479 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:55.672675 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:55.673684 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:55.676065 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:55.687249 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:08:55.694823 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:55.706908 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:55.707818 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:55.750574 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:55.769898 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:55.909193 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:55.936928 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:55.983435 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:55.985342 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:55.987692 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:55.999979 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:56.000217 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:56.041724 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:56.147948 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:56.163466 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:08:56.191957 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 02:08:56.199523 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:56.297205 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:56.297531 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:56.299253 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:56.300555 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:56.311270 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:56.331719 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:56.400658 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:56.445256 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:56.588976 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:08:56.598583 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:08:56.605520 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:56.606349 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:56.607298 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:56.609519 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:56.610816 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:08:56.652876 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:56.664955 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:56.669464 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:56.769377 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 02:08:56.830165 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:56.910535 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:56.911155 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:56.911211 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:56.914525 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:56.926306 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:57.059669 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:57.091483 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:08:57.223340 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:57.224807 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:57.225199 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:57.226095 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:57.227813 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:57.260886 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:57.346689 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:57.386915 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:57.449576 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:57.453187 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:57.457039 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:08:57.534615 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:57.535374 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:57.536860 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:57.551368 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:57.555879 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:57.608888 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:08:57.634843 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:57.684384 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:57.783612 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:57.786456 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:08:57.843923 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:57.845364 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:57.848446 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:57.849010 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:57.849946 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:08:57.865933 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:57.945756 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:08:57.946462 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:58.162606 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:58.164369 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:58.167457 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:58.168011 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:08:58.174593 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:08:58.197250 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:08:58.226351 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:58.235855 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:58.270977 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:58.313408 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:58.335832 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:58.356966 31647 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:08:58.357138 31647 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:08:58.357625 31647 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:08:58.358731 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:58.358901 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:58.358960 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:08:58.359104 31647 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:08:58.456760 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:08:58.484663 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:58.485237 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:58.488127 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:58.488622 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:58.490588 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:58.599684 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:08:58.619457 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:08:58.692174 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:08:58.711256 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:08:58.796074 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:58.796911 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:58.796929 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:58.799397 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:58.801399 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:58.815472 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:58.923110 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:58.949878 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:08:59.045007 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:08:59.105075 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:59.105784 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:59.105906 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:59.119347 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:59.131531 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:59.146627 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:08:59.180804 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:08:59.232916 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:59.317183 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:59.419662 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:59.419723 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:59.419903 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:59.420325 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:59.431669 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:59.524536 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:59.535533 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:08:59.729288 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:08:59.730885 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:08:59.732554 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:08:59.734360 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:08:59.734740 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:08:59.748656 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:08:59.806509 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:08:59.832553 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:08:59.873139 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:08:59.950577 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:08:59.959653 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:09:00.042986 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:00.043849 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:00.044169 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:00.045953 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:00.046310 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:00.056720 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:09:00.291213 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:09:00.295639 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:00.296059 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:09:00.346849 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:00.351233 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:00.353106 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:00.355701 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:00.356866 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:00.382107 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:00.382210 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:09:00.398845 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:00.447618 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:09:00.547829 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:09:00.663625 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:00.663630 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:00.664449 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:00.665755 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:00.674993 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:00.712426 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:09:00.713078 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:09:00.803565 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:00.843788 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:09:00.857717 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:00.861676 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:09:00.915771 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:09:00.943213 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:09:00.964983 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:00.975114 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:00.976670 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:00.976836 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:00.980285 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:00.989743 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:01.084550 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:09:01.179877 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:09:01.280193 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:01.282706 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:01.282869 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:01.282908 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:01.283332 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:01.301607 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:01.314416 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:01.356709 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:09:01.367918 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 02:09:01.426891 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:01.491722 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:01.592340 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:01.594187 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:01.594707 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:01.596048 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:01.596447 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:01.687984 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:09:01.693769 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:09:01.769222 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:01.818758 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 02:09:01.900224 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:01.901902 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:01.902052 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:01.904920 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:01.911438 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:01.911438 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:01.927505 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:01.952030 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:02.162336 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:09:02.217164 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:02.221202 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:02.221315 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:02.227595 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:02.227696 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:02.284577 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:02.319921 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:02.411706 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:02.427870 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:09:02.521903 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:02.522871 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:02.525545 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:02.532884 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:02.547137 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:02.580755 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:09:02.589505 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:09:02.686580 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:02.766738 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:09:02.773780 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:09:02.826200 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:02.837249 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:02.838693 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:02.841878 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:02.847461 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:02.847766 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:02.909495 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:02.921860 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:02.973058 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:03.050917 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:09:03.059641 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240502 02:09:03.142936 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:03.147709 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:03.150298 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:03.150321 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:03.153332 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:03.220958 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:03.226526 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:09:03.264176 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:03.314944 31994 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:09:03.315114 31994 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:09:03.315526 31994 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:09:03.316597 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:03.316696 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:03.316787 8445 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:03.316912 31994 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 02:09:03.449445 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:03.450680 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:03.459939 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:09:03.462227 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:03.464463 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:03.471637 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:03.508926 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:03.513119 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:03.513337 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:09:03.618800 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:03.630333 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:03.666327 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:09:03.757114 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:03.757731 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:03.762719 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:03.763079 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:03.772881 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:03.783826 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:03.852550 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:03.857311 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:03.998898 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:04.063377 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:04.063696 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:04.068550 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:04.069329 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:04.082031 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:04.169427 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:04.217341 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:04.373198 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:04.373278 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:04.374702 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:04.375685 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:04.380625 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:04.385174 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:09:04.388024 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:04.388510 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:04.570850 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:04.678215 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:04.679235 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:09:04.681807 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:04.681823 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:04.683071 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:04.684782 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:04.798673 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:04.957505 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:04.968680 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:04.984907 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:04.985723 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:04.985868 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:04.986232 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:04.988813 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:05.016235 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:05.196252 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:05.233371 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:05.238185 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:05.241322 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:05.279845 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:05.291437 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:05.291536 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:05.292806 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:05.294270 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:05.295660 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:05.349042 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:05.385514 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:05.477891 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:05.535447 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:05.563374 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:05.597566 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:05.599918 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:05.601766 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:05.603215 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:05.622833 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:05.713182 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:05.817529 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:09:05.818163 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:05.875537 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:05.877928 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:05.912966 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:05.913789 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:05.914650 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:05.915174 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:05.917382 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:06.002774 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:06.060634 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:09:06.061224 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:06.110155 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:06.176208 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:06.216584 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:06.217108 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:06.217586 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:06.219745 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:06.221469 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:06.239040 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:06.249152 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:06.264935 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:06.270303 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:06.456884 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:06.467399 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 02:09:06.521270 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:06.523285 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:06.525867 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:06.526312 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:06.548292 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:06.592485 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:09:06.631474 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:06.762188 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:06.788152 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:06.836058 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:06.836839 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:06.837303 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:06.838881 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:06.840003 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:06.847210 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:09:06.917819 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:07.052322 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:07.078723 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:09:07.144369 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:07.151520 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:07.158933 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:07.161374 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:07.166857 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:07.175635 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:07.318441 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:07.336122 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:07.424860 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:07.446341 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:09:07.462375 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:07.463203 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:07.464147 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:07.465421 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:07.475972 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:07.621752 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:07.671329 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:07.770015 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:07.775251 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:07.775573 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:07.776211 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:07.788735 31994 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:09:07.788872 31994 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:09:07.789273 31994 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:09:07.790606 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:07.790771 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:07.790822 8445 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:07.790949 31994 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 02:09:07.794975 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:07.896315 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:07.896376 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:07.954921 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:07.985296 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:07.985956 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:08.014997 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:08.017541 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:08.030315 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:08.033401 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:08.060148 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:09:08.060927 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:08.092619 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:08.092921 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:08.095307 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:08.097730 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:08.107276 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:08.276633 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:08.403122 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:08.404444 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:08.405804 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:08.408444 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:08.416743 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:08.463374 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:08.467767 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:08.510936 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:08.654796 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:09:08.726087 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:08.729561 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:08.729938 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:08.730525 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:08.734591 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:08.814056 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:08.863853 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:08.894160 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:08.904093 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:08.912454 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:09.025094 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:09.035194 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:09.040535 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:09.040781 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:09.046618 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:09.048327 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:09.056728 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:09.064167 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:09.155134 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:09:09.299437 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:09.356000 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:09.359483 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:09.360394 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:09.360778 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:09.376691 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:09.503458 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:09.628253 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:09.637195 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:09:09.673324 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:09.674312 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:09.675696 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:09.679993 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:09.680197 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:09.681077 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:09.713704 348 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:09:09.713841 348 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:09:09.714231 348 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:09:09.715721 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:09.715901 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:09.715955 7117 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:09.716403 348 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:09:09.830945 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:09.882755 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:09.915612 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:09:09.987607 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:09.987751 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:09.990006 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:09.992920 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:10.001667 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:10.068552 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:10.116250 31214 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:09:10.116410 31214 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:09:10.116861 31214 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:09:10.117991 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:10.118127 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:10.118208 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:10.118362 31214 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:09:10.165040 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:10.200177 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:10.212114 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:10.258217 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:09:10.303479 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:10.303896 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:10.306207 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:10.307094 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:10.309496 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:10.410425 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:09:10.480762 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:09:10.544097 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:10.552675 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:09:10.617624 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:10.619779 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:10.620242 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:10.621605 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:10.626282 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:10.638976 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:10.639474 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:10.639688 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:10.690091 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:10.823973 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:09:10.932097 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:10.936486 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:10.937837 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:10.938557 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:10.959009 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:11.031953 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:11.091949 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 02:09:11.130235 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:11.165479 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:09:11.166358 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:09:11.192983 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:11.197484 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:09:11.250586 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:11.259019 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:11.260830 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:11.261240 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:11.262310 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:11.286007 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:11.291057 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:09:11.296746 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:11.306067 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:09:11.375885 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:11.414124 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:11.569970 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:11.572791 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:11.574014 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:11.574714 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:11.582163 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:11.610003 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:11.641939 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 02:09:11.659193 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:09:11.669054 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:11.875510 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 02:09:11.888882 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:11.890513 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:11.891037 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:11.895730 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:11.914949 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:11.936002 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:11.985949 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:09:12.002689 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:12.049968 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:12.057667 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:09:12.205118 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:12.212302 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:12.212363 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:12.220335 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:12.221228 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:12.375248 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:09:12.433069 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:09:12.455757 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:09:12.461930 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:12.469045 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:12.479238 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:09:12.527103 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:12.527487 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:12.529021 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:12.530212 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:12.550979 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:12.567327 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:12.587733 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:09:12.723856 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:09:12.847350 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:12.848035 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:12.852650 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:12.852677 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:12.853011 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:12.899415 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:12.993579 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:13.001423 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:13.008575 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:13.070880 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:09:13.073148 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:13.082435 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:09:13.163439 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:13.163914 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:13.166430 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:13.170629 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:13.181445 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:13.352828 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:09:13.428843 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:13.466025 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:13.476725 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:13.478825 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:13.482582 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:13.483827 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:13.490523 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:13.507967 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:09:13.516638 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:13.532670 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:13.569555 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:13.569554 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:09:13.780666 31994 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:09:13.780820 31994 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:09:13.781288 31994 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:09:13.783102 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:13.783563 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:13.783643 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:13.784618 31994 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:09:13.789814 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:09:13.795068 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:13.796048 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:13.797161 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:13.799750 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:13.802565 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:13.900679 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:13.920852 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:14.000630 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:09:14.084630 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:14.096693 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:14.099821 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:14.101392 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:14.101511 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:14.102744 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:14.165863 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:14.169117 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:09:14.404184 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:14.408036 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:14.410320 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:14.411207 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:14.427054 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:14.437244 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:09:14.463258 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:14.488993 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:14.626384 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:14.631058 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:14.713793 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:14.719048 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:14.720425 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:14.721937 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:14.731536 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:14.793471 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:14.970741 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:09:15.022930 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:15.023412 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:15.023597 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:15.024278 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:15.025943 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:15.061544 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:15.068887 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:15.069754 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:09:15.101817 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:15.148587 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:09:15.156662 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:15.165820 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:15.190690 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:09:15.322072 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:15.322923 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:15.322991 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:15.332476 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:15.339411 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:15.395787 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:15.415695 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:15.555328 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:15.575223 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:15.583153 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:15.626847 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:15.628671 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:15.630162 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:15.630673 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:15.634347 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:15.720734 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:09:15.931533 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:15.931937 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:15.933382 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:15.934758 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:15.948951 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:15.993887 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:15.996405 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:16.059758 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:09:16.087962 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:16.088052 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:16.088977 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:16.123180 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:09:16.184106 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 02:09:16.184722 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:09:16.240013 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:16.240036 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:16.242028 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:16.242697 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:16.242745 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:16.328908 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:16.413182 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:16.456874 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:09:16.463080 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 02:09:16.463632 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:16.536182 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:16.542732 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:16.545413 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:16.549593 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:16.563036 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:16.582993 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:16.647640 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:16.842111 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:09:16.852300 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:16.854528 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:16.855381 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:16.863968 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:16.865460 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:16.928620 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:17.000521 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:17.007859 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:09:17.008505 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:09:17.162541 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:17.163125 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:17.168179 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:17.171298 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:17.175624 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:17.195029 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:17.202788 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240502 02:09:17.238500 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:17.244009 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:09:17.363970 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:17.397037 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:09:17.481945 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:17.481947 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:17.482046 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:17.484858 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:17.487489 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:17.489090 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:17.505290 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:17.596999 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:09:17.628183 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:09:17.684631 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:17.722378 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:09:17.784291 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:17.784317 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:17.792685 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:17.800920 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:17.801416 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:17.811203 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:17.893762 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:17.927815 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:18.032016 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:09:18.097796 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:18.102782 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:18.103528 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:18.104507 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:18.115345 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:18.126345 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:09:18.212464 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:09:18.237010 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:18.289090 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:18.320699 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:18.409430 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:18.410948 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:18.411218 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:18.411489 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:18.411749 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:18.532382 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:09:18.546051 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:18.638446 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:09:18.657116 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:18.705739 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:09:18.712594 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:18.713682 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:18.719970 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:18.721881 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:18.726558 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:18.763967 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:09:18.766721 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:09:18.774173 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:18.775146 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:18.845252 1608 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:09:18.845373 1608 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:09:18.845779 1608 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:09:18.846787 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:18.846884 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:18.846967 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:18.847090 1608 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:09:18.888640 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:18.959555 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:19.019850 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:09:19.028932 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:19.030045 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:19.033452 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:19.041309 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:19.046743 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:19.260438 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:19.321012 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:19.336757 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:19.337121 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:19.338035 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:19.339442 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:19.341389 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:19.347443 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:19.622244 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:09:19.646875 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:19.647456 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:19.651090 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:19.651440 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:19.652165 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:19.697933 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:19.739869 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:09:19.768337 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:19.771466 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:19.822880 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:19.942950 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:09:19.945629 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:19.948695 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:19.951715 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:19.959231 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:09:19.959564 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:19.959900 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:19.982065 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:20.101953 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:20.132371 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:09:20.251495 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:20.252910 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:20.253926 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:20.254846 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:20.268187 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:20.278013 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:20.344364 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:20.347098 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:20.357877 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:20.459093 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:20.516458 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:09:20.572100 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:20.573757 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:20.575548 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:20.577109 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:20.577572 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:20.813560 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:20.857968 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:20.874945 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:20.875607 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:20.878911 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:20.880280 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:20.882514 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:20.896832 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:20.919091 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:21.031567 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:09:21.190277 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:21.192593 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:21.202904 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:09:21.203361 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:21.204046 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:21.214762 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:21.276209 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:21.345304 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 02:09:21.345919 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:09:21.347404 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:09:21.384541 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:21.399147 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:09:21.404263 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:21.411602 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:21.413045 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:21.508719 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:21.512796 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:21.512843 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:21.512967 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:21.515878 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:21.646368 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:09:21.671319 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:09:21.817165 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:21.819856 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:21.819944 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:21.824869 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:21.832883 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:21.840677 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:21.870007 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:21.965582 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:21.989939 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:09:22.042778 6982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 02:09:22.129652 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:22.132043 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:22.134272 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:22.135811 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:22.147176 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:22.154884 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:22.247071 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:22.293854 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:09:22.316223 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:22.346972 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 02:09:22.351917 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:22.443161 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:09:22.443660 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:22.444119 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:22.445026 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:22.446646 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:22.446945 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:22.482661 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:22.529361 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:22.691763 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:09:22.761817 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:22.762238 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:22.762926 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:22.763047 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:22.764458 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:22.813738 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:09:22.917996 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:09:22.929562 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:22.939723 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:22.961032 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:23.069589 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:23.071373 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:23.071666 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:23.073062 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:23.083760 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:23.087390 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:23.170289 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:09:23.215235 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:09:23.339406 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:23.379173 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:23.379354 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:23.379623 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:23.383235 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:23.403972 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:23.460214 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:09:23.505473 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:23.520406 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:23.525381 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:23.624292 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:09:23.646222 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:23.680451 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:09:23.690650 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:23.696307 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:23.696372 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:23.702570 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:23.703171 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:23.725807 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:23.821875 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:09:23.850313 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:09:23.882763 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:23.995350 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:09:24.008344 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:24.010047 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:24.011435 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:24.011598 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:24.028451 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:24.048112 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:24.131836 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:24.317693 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:24.321486 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:24.323033 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:24.324184 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:24.326769 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:24.339781 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:24.507923 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:24.510579 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:24.623634 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:24.626871 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:24.627043 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:24.628156 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:24.629168 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:24.629232 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:24.665035 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:24.713256 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:24.780314 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:24.822145 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:24.879602 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:24.923147 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:09:24.931010 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:24.932227 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:24.934319 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:24.935664 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:24.935779 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:24.944744 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:25.105953 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:25.129115 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:25.132438 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:25.164088 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:09:25.234202 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:25.236032 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:25.236522 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:25.236719 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:25.250005 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:25.272586 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:25.415688 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:25.511651 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:25.538427 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:25.543861 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:25.544158 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:25.549659 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:25.551888 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:25.640589 2903 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:09:25.640709 2903 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:09:25.641084 2903 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:09:25.642063 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:25.642187 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:25.642262 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:25.642414 2903 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:09:25.713493 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:25.790014 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:25.796335 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:09:25.855490 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:25.855885 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:25.856530 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:25.856638 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:25.875289 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:25.924167 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:25.929462 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:25.946714 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:26.003413 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:26.157987 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:09:26.173266 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:26.174289 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:26.174880 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:26.176575 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:26.178670 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:26.181331 2334 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:09:26.181449 2334 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:09:26.181890 2334 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:09:26.185649 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:26.199448 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:26.199570 6849 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:26.199838 2334 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:09:26.250458 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:26.261899 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:26.314893 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:26.324108 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:09:26.333312 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:26.371439 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 02:09:26.381114 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:26.396401 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:26.429201 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:09:26.486863 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:26.489522 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:26.491300 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:26.491665 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:26.492164 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:26.720129 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:26.794591 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:26.796696 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:26.797787 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:26.798069 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:26.802596 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:26.846024 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:26.900882 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:09:26.901473 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:26.964318 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:27.051074 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 02:09:27.062058 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:09:27.095494 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:27.097137 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:27.101169 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:27.101464 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:27.105109 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:27.115796 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:27.314299 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:27.336264 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:27.402091 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:09:27.405586 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:27.406816 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:27.407481 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:27.407514 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:27.409174 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:27.414783 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:27.421543 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:27.454372 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:27.462256 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:09:27.478484 2421 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:09:27.478657 2421 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:09:27.479113 2421 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:09:27.479661 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 02:09:27.480341 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:27.480432 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:27.480525 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:27.480679 2421 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 02:09:27.563912 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:27.573529 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:27.599587 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:27.719542 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:27.720515 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:27.721335 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:27.724495 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:27.747089 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:27.780122 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:27.797067 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:27.944504 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:28.031936 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:28.036810 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:28.038632 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:28.039613 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:28.051970 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:28.060360 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:28.106173 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:28.108198 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:28.260524 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 02:09:28.337052 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:28.343637 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:28.344504 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:28.346033 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:28.346241 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:28.347668 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:28.454962 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:28.531920 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:28.541075 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:28.587903 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:09:28.650333 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:28.653220 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:28.658576 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:28.659417 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:28.671165 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:28.708635 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:28.728024 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:28.840036 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:28.842434 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:28.892453 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:09:28.915184 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:28.958755 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:28.961988 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:28.961988 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:28.962347 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:28.966717 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:28.969911 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:29.056782 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:29.160806 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:09:29.267936 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:29.272289 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:29.272536 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:29.272821 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:29.281949 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:29.288272 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:29.313060 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:29.420075 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:29.471662 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:29.511713 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:29.573340 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:29.574887 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:29.578047 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:29.580967 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:29.580993 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:29.589625 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:29.741313 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:29.776947 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:29.889706 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:29.890303 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:29.892007 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:29.892716 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:29.912166 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:09:29.912999 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:29.948946 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:30.005683 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:30.007273 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:30.103324 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:09:30.134325 3469 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:09:30.134593 3469 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:09:30.135074 3469 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:09:30.136436 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:30.136536 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:30.136673 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:30.136885 3469 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:09:30.189308 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:30.203989 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:30.203984 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:30.209587 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:09:30.212677 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:30.216107 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:30.222873 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:30.351392 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:30.360520 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:30.372020 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:09:30.513674 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:30.513841 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:30.514320 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:30.519099 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:30.521870 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:30.572078 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:30.686297 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:30.786429 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:30.795941 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:30.817445 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:30.819731 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:30.821173 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:30.822773 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:30.836513 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:30.837525 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:30.975312 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:31.069720 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:31.127204 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:31.127982 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:31.130586 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:31.142792 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:31.155583 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:31.168154 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:31.183465 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:31.263417 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:31.265596 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:31.293282 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:31.310086 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:31.396633 6849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 02:09:31.427917 2421 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:09:31.428047 2421 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:09:31.428520 2421 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:09:31.429988 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:31.430279 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:31.430353 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:31.430500 2421 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 02:09:31.439982 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:31.440541 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:31.443405 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:31.443588 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:31.454663 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:31.527572 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:09:31.560032 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:31.615514 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:31.730908 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:31.732010 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:09:31.752775 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:31.755676 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:31.756255 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:31.757416 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:31.765996 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:31.768492 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:31.778533 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:31.827488 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:31.948560 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:09:32.060535 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:32.060693 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:32.063338 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:32.065549 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:32.077538 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:32.109660 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:32.174753 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:32.194075 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:09:32.243364 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 02:09:32.255026 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:32.366626 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:32.369777 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:32.374573 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:32.374847 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:32.381644 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:32.421456 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:32.529158 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 02:09:32.652532 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:32.678336 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:32.679936 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:32.683192 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:32.684401 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:32.696280 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:32.716949 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:32.786926 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:32.792866 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:32.819916 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:32.884251 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:32.965798 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:09:32.995558 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:32.997567 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:32.998311 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:33.002987 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:33.007972 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:33.009356 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:33.009590 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:33.130785 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:33.177773 3946 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:09:33.177928 3946 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:09:33.178452 3946 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:09:33.179596 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:33.179695 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:33.179790 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:33.179951 3946 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:09:33.233935 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:33.252607 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:33.269523 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240502 02:09:33.300575 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:33.301383 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:33.304047 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:33.307500 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:33.311856 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:33.329183 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:33.431608 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:33.526230 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:33.618316 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:33.619267 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:33.620549 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:33.620656 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:33.627997 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:33.658527 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:33.665402 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:33.688954 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:33.714716 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:33.733968 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:33.750077 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:33.879951 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:09:33.929567 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:33.930536 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:33.931780 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:33.932060 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:33.933593 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:33.984123 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:09:34.096534 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:34.109936 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:34.115517 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:34.165632 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:09:34.195345 4077 raft_consensus.cc:482] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:09:34.195495 4077 raft_consensus.cc:504] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:09:34.195924 4077 leader_election.cc:288] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:09:34.197010 8445 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:34.197145 8442 leader_election.cc:334] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:34.197194 8442 leader_election.cc:302] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 954cd49326bf405abd81182d75dc783c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:34.197326 4077 raft_consensus.cc:2726] T 034cf1c3d47a4f7785c16d16815c229d P 954cd49326bf405abd81182d75dc783c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240502 02:09:34.243485 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:34.247926 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:34.254567 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:34.255283 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:34.263928 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:34.274025 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:34.335960 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:34.505107 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:34.542853 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:34.563045 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:34.563320 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:34.563643 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:34.565605 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:34.582358 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:34.587013 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:09:34.655409 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:34.772439 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:09:34.876430 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:34.880010 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:34.880101 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:34.897506 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:34.897642 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:34.965459 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:35.038087 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:35.120157 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:35.196099 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:35.196892 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:35.198709 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:35.213356 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:35.221571 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:35.313802 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:35.367425 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:35.378028 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:09:35.431382 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:35.470991 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:09:35.480443 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:35.511099 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:09:35.512125 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:35.513897 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:35.513934 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:35.515415 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:35.518201 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:35.561271 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:35.564298 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:35.735435 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:09:35.809098 2334 raft_consensus.cc:482] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 070558c278e0457b9cc070ac0d662b5d)
I20240502 02:09:35.809226 2334 raft_consensus.cc:504] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } }
I20240502 02:09:35.809680 2334 leader_election.cc:288] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235), 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475)
W20240502 02:09:35.811372 6849 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:35.811836 6852 leader_election.cc:334] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:35.811909 6852 leader_election.cc:302] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7bcff2fef614e7a9981554c9e6db3c8; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:35.812530 2334 raft_consensus.cc:2726] T 9f6c07e3923d47f29ef32561d7f438e1 P e7bcff2fef614e7a9981554c9e6db3c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:09:35.833837 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:35.836719 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:35.836948 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:35.838595 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:35.848924 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:09:35.940450 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:35.984565 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:35.996052 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:36.005065 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:36.147675 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:36.153633 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:36.154029 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:36.155555 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:36.155808 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:36.173126 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:36.203073 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:09:36.302194 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:09:36.335613 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:36.359354 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:36.424755 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:09:36.425094 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:36.439707 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:09:36.466339 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:36.467864 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:36.468235 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:36.469736 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:36.471130 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:36.496820 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:09:36.497598 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:09:36.567153 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:36.606686 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:36.628963 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:09:36.777434 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:36.780643 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:36.781140 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:36.781183 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:36.799347 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:36.832296 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:36.913007 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:09:37.070250 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:37.085407 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:37.086623 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:37.088109 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:37.097623 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:37.099041 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:37.109573 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:37.173436 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:37.213945 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:37.222098 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:37.256152 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 02:09:37.391016 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:37.394029 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:37.396252 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:37.398615 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:37.408980 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:37.428123 6721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:09:37.432639 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:37.519448 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:37.554636 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:37.620175 4676 raft_consensus.cc:482] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4f53d80e39be410d92d4766692aff1c6)
I20240502 02:09:37.620371 4676 raft_consensus.cc:504] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "4f53d80e39be410d92d4766692aff1c6" member_type: VOTER last_known_addr { host: "127.27.111.198" port: 46475 } } peers { permanent_uuid: "070558c278e0457b9cc070ac0d662b5d" member_type: VOTER last_known_addr { host: "127.27.111.200" port: 37235 } }
I20240502 02:09:37.621274 4676 leader_election.cc:288] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475), 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235)
W20240502 02:09:37.622156 7117 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:37.622346 7114 leader_election.cc:334] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:37.622421 7114 leader_election.cc:302] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86bc8001c3464b929cd5c873861baa0c; no voters: 070558c278e0457b9cc070ac0d662b5d, 4f53d80e39be410d92d4766692aff1c6
I20240502 02:09:37.622574 4676 raft_consensus.cc:2726] T c8c237c50773494abf8813494be0f184 P 86bc8001c3464b929cd5c873861baa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:09:37.697101 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:37.700268 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:37.701349 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:37.704270 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:37.714469 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:37.823733 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240502 02:09:37.833070 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:37.889475 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:37.889915 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:09:37.900625 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:09:37.937135 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:09:37.939114 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:09:38.012766 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:38.013108 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:38.013425 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:38.017249 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:38.018878 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:38.064417 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:38.161638 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:38.214433 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:38.301789 7117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:09:38.303290 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:09:38.323207 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:38.325536 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:38.334939 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:38.335119 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:38.350629 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:38.359117 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:09:38.502089 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:38.624181 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:09:38.637419 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:38.638734 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:38.644209 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:38.646548 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:38.652243 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:38.714341 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:38.763759 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:38.809866 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:38.812748 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:38.821200 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:09:38.834708 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:38.860203 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:09:38.949991 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:38.950850 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:38.953892 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:09:38.954135 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:38.954828 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:38.956784 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:38.978425 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:38.993831 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:39.141707 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:39.203612 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:39.231452 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:09:39.276300 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:39.276715 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:39.283001 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:39.291913 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:39.294442 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:39.400915 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:39.581228 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:39.581247 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:39.585649 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:39.585820 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:39.592229 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:09:39.602602 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:39.615602 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:39.658442 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:39.760134 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:09:39.867168 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:09:39.885049 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:09:39.893478 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:39.896482 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:39.898185 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:39.898900 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:39.906163 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:40.024490 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:40.034901 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:40.111492 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:40.198029 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:40.202083 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:40.202554 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:40.202883 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:40.218603 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:40.277853 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:40.314576 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:40.398104 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:09:40.488679 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:40.509580 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:40.509769 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:40.512740 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:40.514000 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:40.519752 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:09:40.521201 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:40.551156 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:40.585368 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:40.635174 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:40.635299 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:40.654002 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:40.812914 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:40.815655 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:40.816455 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:40.827078 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:40.828083 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:40.963605 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:41.044226 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:41.111718 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:41.116367 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:41.119832 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:41.130539 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:41.130813 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:41.131278 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:41.162767 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:41.238472 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 02:09:41.253151 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:41.319716 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:09:41.424021 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:09:41.439054 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:41.439241 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:41.440150 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:41.442708 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:41.442708 6982 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:09:41.451128 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:41.465442 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:41.477139 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:09:41.485952 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:09:41.534911 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:09:41.592213 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:41.617039 6852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:09:41.741917 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:41.751013 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:41.752707 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:41.757148 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:41.759824 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:41.765161 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:09:41.884615 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:41.987270 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:42.052960 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:42.054543 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:42.055861 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:42.056264 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:42.057389 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:42.202834 7114 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 02:09:42.267802 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:42.276724 6985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 02:09:42.277318 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:42.347189 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:09:42.360546 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:42.360674 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:42.361342 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:42.367421 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:42.375505 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:42.478379 6718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 02:09:42.479332 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:42.585961 6985 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:09:42.671222 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:42.672250 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:42.673056 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:42.673134 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:42.674057 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:42.684650 6849 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 02:09:42.690361 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:09:42.749809 8445 consensus_peers.cc:485] T 17235e019e644b258266636fb6be6648 P 954cd49326bf405abd81182d75dc783c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 02:09:42.794556 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:42.872135 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 02:09:42.892139 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:42.958415 7114 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:09:42.974707 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:42.977871 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:42.978155 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:42.980762 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:42.990128 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 02:09:43.024390 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:43.055053 6852 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:09:43.101497 6982 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:09:43.178892 7117 consensus_peers.cc:485] T a66b932d884e4df8a78841103d88ded0 P 86bc8001c3464b929cd5c873861baa0c -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:09:43.234824 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:43.246914 8445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 02:09:43.275713 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:43.276244 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:43.277228 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:43.278708 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:43.282480 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:43.352609 7114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 02:09:43.353721 6985 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 02:09:43.464529 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:43.466121 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
W20240502 02:09:43.579237 6849 consensus_peers.cc:485] T 39d9e48f67f34ebe850fc388ecd94ca3 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:09:43.579522 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:43.579949 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:43.582727 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:43.585747 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:43.595495 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:43.647889 7114 consensus_peers.cc:462] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
W20240502 02:09:43.670470 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:09:43.893865 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:43.894356 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:43.897426 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:43.898545 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:43.910979 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 02:09:43.929841 7114 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 02:09:43.929945 7114 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:09:43.996845 6852 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 02:09:44.039620 6849 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Couldn't send request to peer 070558c278e0457b9cc070ac0d662b5d. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 02:09:44.042191 5526 raft_consensus.cc:998] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5: Attempting to remove follower 4f53d80e39be410d92d4766692aff1c6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4f53d80e39be410d92d4766692aff1c6 for more than 300 seconds (300.108s)
I20240502 02:09:44.042568 5526 consensus_queue.cc:237] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.043704 6785 raft_consensus.cc:1261] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:09:44.044065 5687 consensus_queue.cc:1035] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:09:44.045643 5526 raft_consensus.cc:2932] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.046365 6785 raft_consensus.cc:2932] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.049008 6640 catalog_manager.cc:5484] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 7 to 8, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
W20240502 02:09:44.052613 6630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 02:09:44.053426 6630 catalog_manager.cc:4573] TS 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e0e05f96dbc542eb999c16dcc0620666: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111)
I20240502 02:09:44.055744 7050 consensus_queue.cc:237] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:09:44.056783 6785 raft_consensus.cc:1261] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 02:09:44.057094 6985 consensus_peers.cc:485] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0e05f96dbc542eb999c16dcc0620666. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.057302 5526 consensus_queue.cc:1035] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:09:44.058856 5687 raft_consensus.cc:2932] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.059350 5526 raft_consensus.cc:998] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5: Attempting to remove follower 070558c278e0457b9cc070ac0d662b5d from the Raft config. Reason: Leader has been unable to successfully communicate with peer 070558c278e0457b9cc070ac0d662b5d for more than 300 seconds (300.120s)
I20240502 02:09:44.059252 6783 raft_consensus.cc:2932] T e0e05f96dbc542eb999c16dcc0620666 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.059643 5526 consensus_queue.cc:237] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:09:44.060695 8507 raft_consensus.cc:1261] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:09:44.061051 5526 consensus_queue.cc:1035] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:09:44.061077 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e0e05f96dbc542eb999c16dcc0620666 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:09:44.061856 6640 catalog_manager.cc:5484] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 8 to 9, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.064083 5526 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.065037 8507 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.066603 6641 catalog_manager.cc:5484] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.069216 7050 consensus_queue.cc:237] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.070152 8507 raft_consensus.cc:1261] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 02:09:44.070327 6984 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): Couldn't send request to peer 85e78945910a4e1abfa101d2edc3644e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.070521 5526 consensus_queue.cc:1035] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:09:44.071833 5687 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.072263 8507 raft_consensus.cc:2932] T 7345f51bd6f84b3880838c2460bafb75 P 954cd49326bf405abd81182d75dc783c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.073619 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7345f51bd6f84b3880838c2460bafb75 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:09:44.074331 6641 catalog_manager.cc:5484] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
W20240502 02:09:44.082242 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:44.162416 5526 raft_consensus.cc:998] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5: Attempting to remove follower 4f53d80e39be410d92d4766692aff1c6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4f53d80e39be410d92d4766692aff1c6 for more than 300 seconds (300.030s)
I20240502 02:09:44.162834 5526 consensus_queue.cc:237] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:09:44.164076 6917 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8 [term 3 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 02:09:44.164485 5687 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 02:09:44.167598 5526 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.169183 6917 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.170413 6640 catalog_manager.cc:5484] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 11 to 12, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New cstate: current_term: 3 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.173728 7050 consensus_queue.cc:237] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 6, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:09:44.175108 6917 raft_consensus.cc:1261] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8 [term 3 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
W20240502 02:09:44.175273 6985 consensus_peers.cc:485] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91db5c1b3d2f4d22b5882a61ea0395c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.175473 5526 consensus_queue.cc:1035] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 02:09:44.176973 5687 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.177474 6917 raft_consensus.cc:2932] T 91db5c1b3d2f4d22b5882a61ea0395c7 P e7bcff2fef614e7a9981554c9e6db3c8 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.178330 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 91db5c1b3d2f4d22b5882a61ea0395c7 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:09:44.180112 6640 catalog_manager.cc:5484] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 12 to 13, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 3 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.191361 5686 raft_consensus.cc:998] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c: Attempting to remove follower 070558c278e0457b9cc070ac0d662b5d from the Raft config. Reason: Leader has been unable to successfully communicate with peer 070558c278e0457b9cc070ac0d662b5d for more than 300 seconds (300.348s)
I20240502 02:09:44.191771 5686 consensus_queue.cc:237] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.193120 6783 raft_consensus.cc:1261] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:09:44.193596 5069 consensus_queue.cc:1035] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:09:44.197362 5069 raft_consensus.cc:2932] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.199050 6783 raft_consensus.cc:2932] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.199860 6635 catalog_manager.cc:5484] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.208649 7181 consensus_queue.cc:237] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
W20240502 02:09:44.210095 7117 consensus_peers.cc:485] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c -> Peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): Couldn't send request to peer e7bcff2fef614e7a9981554c9e6db3c8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.210171 6783 raft_consensus.cc:1261] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:09:44.210644 5736 consensus_queue.cc:1035] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:09:44.212333 5686 raft_consensus.cc:2932] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.212745 6783 raft_consensus.cc:2932] T 8529a93abd6e4d62bbc0bddcf2fda78a P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.214183 6629 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8529a93abd6e4d62bbc0bddcf2fda78a with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:09:44.215134 6635 catalog_manager.cc:5484] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 8 to 9, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
W20240502 02:09:44.217855 6721 consensus_peers.cc:485] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198:46475): Couldn't send request to peer 4f53d80e39be410d92d4766692aff1c6. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.198:46475: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 02:09:44.228497 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:44.228722 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:44.229652 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:44.230602 8489 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:44.248481 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:44.293819 5749 raft_consensus.cc:998] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8: Attempting to remove follower 4f53d80e39be410d92d4766692aff1c6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4f53d80e39be410d92d4766692aff1c6 for more than 300 seconds (300.085s)
I20240502 02:09:44.294180 5749 consensus_queue.cc:237] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } }
I20240502 02:09:44.295362 7050 raft_consensus.cc:1261] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 02:09:44.295733 2334 consensus_queue.cc:1035] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:09:44.297273 5749 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 02:09:44.297631 7050 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 02:09:44.299618 6637 catalog_manager.cc:5484] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 9 to 10, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } }
I20240502 02:09:44.303452 5526 raft_consensus.cc:998] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5: Attempting to remove follower 4f53d80e39be410d92d4766692aff1c6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4f53d80e39be410d92d4766692aff1c6 for more than 300 seconds (300.066s)
I20240502 02:09:44.303754 5526 consensus_queue.cc:237] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 02:09:44.304925 7181 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 02:09:44.305307 5687 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 02:09:44.305964 6917 consensus_queue.cc:237] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:09:44.307107 7050 raft_consensus.cc:1261] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 02:09:44.307255 5687 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 02:09:44.307484 2334 consensus_queue.cc:1035] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:09:44.307654 7181 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
W20240502 02:09:44.309108 6852 consensus_peers.cc:485] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 818dbc9a8e054eeaa122e7919a311806. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.309705 6637 catalog_manager.cc:5484] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 11 to 12, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 02:09:44.310020 5749 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.311203 7050 raft_consensus.cc:2932] T 818dbc9a8e054eeaa122e7919a311806 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.311661 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 818dbc9a8e054eeaa122e7919a311806 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:09:44.312538 6641 catalog_manager.cc:5484] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 10 to 11, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.313490 7050 consensus_queue.cc:237] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
W20240502 02:09:44.315485 6984 consensus_peers.cc:485] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): Couldn't send request to peer 85e78945910a4e1abfa101d2edc3644e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.316049 7181 raft_consensus.cc:1261] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 02:09:44.316399 5526 consensus_queue.cc:1035] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 02:09:44.317687 5687 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.318025 7181 raft_consensus.cc:2932] T 47a276e9780e448a9b3c6eab67925514 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.319290 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 47a276e9780e448a9b3c6eab67925514 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:09:44.319680 6635 catalog_manager.cc:5484] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 12 to 13, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.332489 5686 raft_consensus.cc:998] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c: Attempting to remove follower 070558c278e0457b9cc070ac0d662b5d from the Raft config. Reason: Leader has been unable to successfully communicate with peer 070558c278e0457b9cc070ac0d662b5d for more than 300 seconds (300.251s)
I20240502 02:09:44.332844 5686 consensus_queue.cc:237] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:09:44.334123 8507 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 02:09:44.334548 5736 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:09:44.336719 5686 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.337201 8507 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.339438 6641 catalog_manager.cc:5484] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 8 to 9, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New cstate: current_term: 2 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.342705 7181 consensus_queue.cc:237] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.343856 8507 raft_consensus.cc:1261] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240502 02:09:44.344018 7116 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): Couldn't send request to peer 85e78945910a4e1abfa101d2edc3644e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.344259 5736 consensus_queue.cc:1035] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:09:44.345650 5686 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.346022 8507 raft_consensus.cc:2932] T 44c55b199a24432e902cce9121fc0780 P 954cd49326bf405abd81182d75dc783c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.346971 6629 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 44c55b199a24432e902cce9121fc0780 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:09:44.348038 6641 catalog_manager.cc:5484] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 9 to 10, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New cstate: current_term: 2 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.372776 5736 raft_consensus.cc:998] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c: Attempting to remove follower 070558c278e0457b9cc070ac0d662b5d from the Raft config. Reason: Leader has been unable to successfully communicate with peer 070558c278e0457b9cc070ac0d662b5d for more than 300 seconds (300.184s)
I20240502 02:09:44.373195 5736 consensus_queue.cc:237] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.374468 6783 raft_consensus.cc:1261] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:09:44.374835 5686 consensus_queue.cc:1035] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:09:44.376428 5736 raft_consensus.cc:2932] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.376830 6783 raft_consensus.cc:2932] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.379094 6635 catalog_manager.cc:5484] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.382081 7181 consensus_queue.cc:237] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:09:44.383086 6783 raft_consensus.cc:1261] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 02:09:44.383328 7117 consensus_peers.cc:485] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c -> Peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): Couldn't send request to peer e7bcff2fef614e7a9981554c9e6db3c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4154be3f1f3c428580463c419d69892e. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.383430 5686 consensus_queue.cc:1035] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:09:44.384073 5736 raft_consensus.cc:998] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c: Attempting to remove follower 070558c278e0457b9cc070ac0d662b5d from the Raft config. Reason: Leader has been unable to successfully communicate with peer 070558c278e0457b9cc070ac0d662b5d for more than 300 seconds (300.207s)
I20240502 02:09:44.384337 5736 consensus_queue.cc:237] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.385325 6783 raft_consensus.cc:1261] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:09:44.385295 5736 raft_consensus.cc:2932] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.385748 5069 consensus_queue.cc:1035] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:09:44.385689 6783 raft_consensus.cc:2932] T 4154be3f1f3c428580463c419d69892e P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.386565 6629 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4154be3f1f3c428580463c419d69892e with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:09:44.387979 6635 catalog_manager.cc:5484] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 8 to 9, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.388007 5736 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.388443 6785 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.391635 6641 catalog_manager.cc:5484] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 7 to 8, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.394357 7181 consensus_queue.cc:237] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:09:44.395406 6785 raft_consensus.cc:1261] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 02:09:44.395790 7117 consensus_peers.cc:485] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c -> Peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305): Couldn't send request to peer e7bcff2fef614e7a9981554c9e6db3c8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.395960 5736 consensus_queue.cc:1035] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:09:44.397279 5069 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.397711 6785 raft_consensus.cc:2932] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.398506 6629 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7fa358140eb742b587a4d6a203738b4f with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:09:44.399664 6641 catalog_manager.cc:5484] T 7fa358140eb742b587a4d6a203738b4f P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 8 to 9, VOTER e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.456867 5749 raft_consensus.cc:998] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8: Attempting to remove follower 4f53d80e39be410d92d4766692aff1c6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4f53d80e39be410d92d4766692aff1c6 for more than 300 seconds (300.146s)
I20240502 02:09:44.457347 5749 consensus_queue.cc:237] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 02:09:44.458668 7181 raft_consensus.cc:1261] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:09:44.459100 5760 consensus_queue.cc:1035] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:09:44.460809 5749 raft_consensus.cc:2932] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 02:09:44.461258 7181 raft_consensus.cc:2932] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 02:09:44.463507 6635 catalog_manager.cc:5484] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 7 to 8, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 02:09:44.466928 6917 consensus_queue.cc:237] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
W20240502 02:09:44.468434 6851 consensus_peers.cc:485] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): Couldn't send request to peer 85e78945910a4e1abfa101d2edc3644e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed7b67af7b1c4255a7c2377570b58e70. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.468494 7181 raft_consensus.cc:1261] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:09:44.468894 5749 consensus_queue.cc:1035] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:09:44.469599 5763 ts_tablet_manager.cc:902] T e0e05f96dbc542eb999c16dcc0620666 P 954cd49326bf405abd81182d75dc783c: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
F20240502 02:09:44.470141 5763 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.queue_time_us, tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.queue_time_us) (tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.run_cpu_time_us, tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.run_cpu_time_us) (tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.run_wall_time_us, tablet-download-pool-034cf1c3d47a4f7785c16d16815c229d.run_wall_time_us) (tablet-download-pool-0359dfbfd11746f1bf62a465f14634ec.queue_time_us, tablet-download-pool-0359dfbfd11746f1bf62a465f14634ec.queue_time_us) (tablet-download-pool-0359dfbfd11746f1bf62a465f14634ec.run_cpu_time_us, tablet-download-pool-0359dfbfd11746f1bf62a465f14634ec.run_cpu_time_us) (tablet-download-pool-0359dfbfd11746f1bf62a465f14634ec.run_wall_time_us, tablet-download-pool-0359dfbfd11746f1bf62a465f14634ec.run_wall_time_us) (tablet-download-pool-17235e019e644b258266636fb6be6648.queue_time_us, tablet-download-pool-17235e019e644b258266636fb6be6648.queue_time_us) (tablet-download-pool-17235e019e644b258266636fb6be6648.run_cpu_time_us, tablet-download-pool-17235e019e644b258266636fb6be6648.run_cpu_time_us) (tablet-download-pool-17235e019e644b258266636fb6be6648.run_wall_time_us, tablet-download-pool-17235e019e644b258266636fb6be6648.run_wall_time_us) (tablet-download-pool-1c54b912f802461c8cac092ac3222c87.queue_time_us, tablet-download-pool-1c54b912f802461c8cac092ac3222c87.queue_time_us) (tablet-download-pool-1c54b912f802461c8cac092ac3222c87.run_cpu_time_us, tablet-download-pool-1c54b912f802461c8cac092ac3222c87.run_cpu_time_us) (tablet-download-pool-1c54b912f802461c8cac092ac3222c87.run_wall_time_us, tablet-download-pool-1c54b912f802461c8cac092ac3222c87.run_wall_time_us) (tablet-download-pool-44c55b199a24432e902cce9121fc0780.queue_time_us, tablet-download-pool-44c55b199a24432e902cce9121fc0780.queue_time_us) (tablet-download-pool-44c55b199a24432e902cce9121fc0780.run_cpu_time_us, tablet-download-pool-44c55b199a24432e902cce9121fc0780.run_cpu_time_us) (tablet-download-pool-44c55b199a24432e902cce9121fc0780.run_wall_time_us, tablet-download-pool-44c55b199a24432e902cce9121fc0780.run_wall_time_us) (tablet-download-pool-47a276e9780e448a9b3c6eab67925514.queue_time_us, tablet-download-pool-47a276e9780e448a9b3c6eab67925514.queue_time_us) (tablet-download-pool-47a276e9780e448a9b3c6eab67925514.run_cpu_time_us, tablet-download-pool-47a276e9780e448a9b3c6eab67925514.run_cpu_time_us) (tablet-download-pool-47a276e9780e448a9b3c6eab67925514.run_wall_time_us, tablet-download-pool-47a276e9780e448a9b3c6eab67925514.run_wall_time_us) (tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.queue_time_us, tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.queue_time_us) (tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.run_cpu_time_us, tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.run_cpu_time_us) (tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.run_wall_time_us, tablet-download-pool-48eeb721a60c4be888041bd31cfeab7a.run_wall_time_us) (tablet-download-pool-4e44267bd266407f9b72c66e3d3579bb.queue_time_us, tablet-download-pool-4e44267bd266407f9b72c66e3d3579bb.queue_time_us) (tablet-download-pool-4e44267bd266407f9b72c66e3d3579bb.run_cpu_time_us, tablet-download-pool-4e44267bd266407f9b72c66e3d3579bb.run_cpu_time_us) (tablet-download-pool-4e44267bd266407f9b72c66e3d3579bb.run_wall_time_us, tablet-download-pool-4e44267bd266407f9b72c66e3d3579bb.run_wall_time_us) (tablet-download-pool-7345f51bd6f84b3880838c2460bafb75.queue_time_us, tablet-download-pool-7345f51bd6f84b3880838c2460bafb75.queue_time_us) (tablet-download-pool-7345f51bd6f84b3880838c2460bafb75.run_cpu_time_us, tablet-download-pool-7345f51bd6f84b3880838c2460bafb75.run_cpu_time_us) (tablet-download-pool-7345f51bd6f84b3880838c2460bafb75.run_wall_time_us, tablet-download-pool-7345f51bd6f84b3880838c2460bafb75.run_wall_time_us) (tablet-download-pool-9313079c48c94ab1b7e022a91befafcd.queue_time_us, tablet-download-pool-9313079c48c94ab1b7e022a91befafcd.queue_time_us) (tablet-download-pool-9313079c48c94ab1b7e022a91befafcd.run_cpu_time_us, tablet-download-pool-9313079c48c94ab1b7e022a91befafcd.run_cpu_time_us) (tablet-download-pool-9313079c48c94ab1b7e022a91befafcd.run_wall_time_us, tablet-download-pool-9313079c48c94ab1b7e022a91befafcd.run_wall_time_us) (tablet-download-pool-9ecbf1c8ee8c46b195d2374a5db81ac0.queue_time_us, tablet-download-pool-9ecbf1c8ee8c46b195d2374a5db81ac0.queue_time_us) (tablet-download-pool-9ecbf1c8ee8c46b195d2374a5db81ac0.run_cpu_time_us, tablet-download-pool-9ecbf1c8ee8c46b195d2374a5db81ac0.run_cpu_time_us) (tablet-download-pool-9ecbf1c8ee8c46b195d2374a5db81ac0.run_wall_time_us, tablet-download-pool-9ecbf1c8ee8c46b195d2374a5db81ac0.run_wall_time_us) (tablet-download-pool-b69d6e907b3b400f93721981732c006b.queue_time_us, tablet-download-pool-b69d6e907b3b400f93721981732c006b.queue_time_us) (tablet-download-pool-b69d6e907b3b400f93721981732c006b.run_cpu_time_us, tablet-download-pool-b69d6e907b3b400f93721981732c006b.run_cpu_time_us) (tablet-download-pool-b69d6e907b3b400f93721981732c006b.run_wall_time_us, tablet-download-pool-b69d6e907b3b400f93721981732c006b.run_wall_time_us) (tablet-download-pool-d4d429fe60e74108bc0c9ca8509365f2.queue_time_us, tablet-download-pool-d4d429fe60e74108bc0c9ca8509365f2.queue_time_us) (tablet-download-pool-d4d429fe60e74108bc0c9ca8509365f2.run_cpu_time_us, tablet-download-pool-d4d429fe60e74108bc0c9ca8509365f2.run_cpu_time_us) (tablet-download-pool-d4d429fe60e74108bc0c9ca8509365f2.run_wall_time_us, tablet-download-pool-d4d429fe60e74108bc0c9ca8509365f2.run_wall_time_us) (tablet-download-pool-e0e05f96dbc542eb999c16dcc0620666.queue_time_us, tablet-download-pool-e0e05f96dbc542eb999c16dcc0620666.queue_time_us) (tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.queue_time_us, tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.queue_time_us) (tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.run_cpu_time_us, tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.run_cpu_time_us) (tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.run_wall_time_us, tablet-download-pool-e6ac26fa6de44f7ca68e6c0b6b959db7.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240502 02:09:44.470579 5749 raft_consensus.cc:2932] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
*** Check failure stack trace: ***
I20240502 02:09:44.471179 7181 raft_consensus.cc:2932] T ed7b67af7b1c4255a7c2377570b58e70 P 86bc8001c3464b929cd5c873861baa0c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.472683 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ed7b67af7b1c4255a7c2377570b58e70 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
@ 0x7f01d7d1bdcd google::LogMessage::Fail() at ??:0
I20240502 02:09:44.474115 6635 catalog_manager.cc:5484] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 8 to 9, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
@ 0x7f01d7d1fb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f01d7d1b7cc google::LogMessage::Flush() at ??:0
@ 0x7f01d7d1cf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f01d9185044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f01d9169f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f01d9167efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f01dbd1a77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f01dbd23a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f01dbda476f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f01dbdb84f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f01dbda3226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f01dbdb2c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55f3d0c7e75c std::function<>::operator()() at ??:0
I20240502 02:09:44.516597 5686 raft_consensus.cc:998] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c: Attempting to remove follower 070558c278e0457b9cc070ac0d662b5d from the Raft config. Reason: Leader has been unable to successfully communicate with peer 070558c278e0457b9cc070ac0d662b5d for more than 300 seconds (300.498s)
I20240502 02:09:44.517043 5686 consensus_queue.cc:237] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:09:44.518429 6917 raft_consensus.cc:1261] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
@ 0x7f01d916ce06 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 02:09:44.519480 5069 consensus_queue.cc:1035] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
@ 0x7f01d916d6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f01d916f03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55f3d0c7e75c std::function<>::operator()() at ??:0
I20240502 02:09:44.527053 5069 raft_consensus.cc:2932] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
@ 0x7f01d915c807 kudu::Thread::SuperviseThread() at ??:0
I20240502 02:09:44.529088 6917 raft_consensus.cc:2932] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.529956 6641 catalog_manager.cc:5484] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c reported cstate change: config changed from index 5 to 6, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
@ 0x7f01d999a6db start_thread at ??:0
@ 0x7f01d6ed271f clone at ??:0
I20240502 02:09:44.537865 7181 consensus_queue.cc:237] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.539219 5806 ts_tablet_manager.cc:902] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 02:09:44.539426 6917 raft_consensus.cc:1261] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc8001c3464b929cd5c873861baa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 02:09:44.539479 7116 consensus_peers.cc:485] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c -> Peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): Couldn't send request to peer 85e78945910a4e1abfa101d2edc3644e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.540082 5686 consensus_queue.cc:1035] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:09:44.540406 5806 tablet_copy_client.cc:308] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 02:09:44.541219 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 7345f51bd6f84b3880838c2460bafb75 from peer 85e78945910a4e1abfa101d2edc3644e ({username='slave'} at 127.27.111.193:42103)
I20240502 02:09:44.541325 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 7345f51bd6f84b3880838c2460bafb75 from peer 85e78945910a4e1abfa101d2edc3644e at {username='slave'} at 127.27.111.193:42103: session id = 85e78945910a4e1abfa101d2edc3644e-7345f51bd6f84b3880838c2460bafb75
I20240502 02:09:44.541893 5736 raft_consensus.cc:2932] T 347b161e9ec84640a0668836ce9e29eb P 86bc8001c3464b929cd5c873861baa0c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.542546 7070 tablet_copy_source_session.cc:205] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:09:44.542812 6917 raft_consensus.cc:2932] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.551610 6629 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 347b161e9ec84640a0668836ce9e29eb with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:09:44.553416 6639 catalog_manager.cc:5484] T 347b161e9ec84640a0668836ce9e29eb P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 6 to 7, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New cstate: current_term: 1 leader_uuid: "86bc8001c3464b929cd5c873861baa0c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
W20240502 02:09:44.554109 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:44.557376 5806 ts_tablet_manager.cc:1875] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:09:44.560191 5806 ts_tablet_manager.cc:1888] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 02:09:44.560371 5806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7345f51bd6f84b3880838c2460bafb75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:44.561673 5806 tablet_copy_client.cc:783] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 5 data blocks...
I20240502 02:09:44.568333 5687 raft_consensus.cc:998] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5: Attempting to remove follower 4f53d80e39be410d92d4766692aff1c6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4f53d80e39be410d92d4766692aff1c6 for more than 300 seconds (300.288s)
I20240502 02:09:44.568684 5687 consensus_queue.cc:237] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:09:44.572516 5760 raft_consensus.cc:998] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8: Attempting to remove follower 070558c278e0457b9cc070ac0d662b5d from the Raft config. Reason: Leader has been unable to successfully communicate with peer 070558c278e0457b9cc070ac0d662b5d for more than 300 seconds (300.305s)
I20240502 02:09:44.572860 5760 consensus_queue.cc:237] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.574087 6785 raft_consensus.cc:1261] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 02:09:44.574539 5749 consensus_queue.cc:1035] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:09:44.575783 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:44.579617 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:09:44.580540 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:44.591089 5639 raft_consensus.cc:998] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5: Attempting to remove follower 070558c278e0457b9cc070ac0d662b5d from the Raft config. Reason: Leader has been unable to successfully communicate with peer 070558c278e0457b9cc070ac0d662b5d for more than 300 seconds (300.295s)
I20240502 02:09:44.591445 5639 consensus_queue.cc:237] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.591507 5749 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.592934 6785 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 02:09:44.593310 5526 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 02:09:44.594192 6785 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.594671 6637 catalog_manager.cc:5484] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 9 to 10, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.595528 5806 tablet_copy_client.cc:647] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 1 WAL segments...
I20240502 02:09:44.597606 5806 tablet_copy_client.cc:523] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:09:44.599129 6917 consensus_queue.cc:237] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 02:09:44.600821 6785 raft_consensus.cc:1261] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Refusing update from remote peer e7bcff2fef614e7a9981554c9e6db3c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 02:09:44.601176 5806 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
W20240502 02:09:44.601179 6851 consensus_peers.cc:485] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569): Couldn't send request to peer 86bc8001c3464b929cd5c873861baa0c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.601287 2334 consensus_queue.cc:1035] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:09:44.602849 2334 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 02:09:44.604947 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet dcc5daef41bc44c4855f2bcb44e1c6d5 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:09:44.606950 6641 catalog_manager.cc:5484] T dcc5daef41bc44c4855f2bcb44e1c6d5 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 10 to 11, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New cstate: current_term: 1 leader_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 02:09:44.609810 6785 raft_consensus.cc:2932] T dcc5daef41bc44c4855f2bcb44e1c6d5 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 86bc8001c3464b929cd5c873861baa0c (127.27.111.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } }
I20240502 02:09:44.610558 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:44.614805 5526 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.615859 5686 raft_consensus.cc:998] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c: Attempting to remove follower 4f53d80e39be410d92d4766692aff1c6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4f53d80e39be410d92d4766692aff1c6 for more than 300 seconds (300.299s)
I20240502 02:09:44.616175 5686 consensus_queue.cc:237] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } }
I20240502 02:09:44.616879 6785 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.620554 6641 catalog_manager.cc:5484] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e reported cstate change: config changed from index 12 to 13, VOTER 070558c278e0457b9cc070ac0d662b5d (127.27.111.200) evicted. New cstate: current_term: 2 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.623663 5806 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:09:44.623919 5806 tablet_bootstrap.cc:492] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap complete.
I20240502 02:09:44.624100 5806 ts_tablet_manager.cc:1367] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.004s
I20240502 02:09:44.624497 5806 raft_consensus.cc:348] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.624591 5806 raft_consensus.cc:729] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 02:09:44.625891 7050 consensus_queue.cc:237] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
I20240502 02:09:44.627007 6785 raft_consensus.cc:1261] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 02:09:44.627421 5687 consensus_queue.cc:1035] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 02:09:44.628706 5806 consensus_queue.cc:260] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.628824 5687 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.629002 5806 ts_tablet_manager.cc:1398] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 02:09:44.629410 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 85e78945910a4e1abfa101d2edc3644e-7345f51bd6f84b3880838c2460bafb75 received from {username='slave'} at 127.27.111.193:42103
I20240502 02:09:44.629473 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 85e78945910a4e1abfa101d2edc3644e-7345f51bd6f84b3880838c2460bafb75 on tablet 7345f51bd6f84b3880838c2460bafb75 with peer 85e78945910a4e1abfa101d2edc3644e
I20240502 02:09:44.630471 6785 raft_consensus.cc:2932] T a0548274c47148d9b611fd3c978a0c34 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.630719 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a0548274c47148d9b611fd3c978a0c34 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:09:44.632728 6637 catalog_manager.cc:5484] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 13 to 14, VOTER 954cd49326bf405abd81182d75dc783c (127.27.111.197) added. New cstate: current_term: 2 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } }
I20240502 02:09:44.638470 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling UndoDeltaBlockGCOp(7345f51bd6f84b3880838c2460bafb75): 4834 bytes on disk
I20240502 02:09:44.638911 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: UndoDeltaBlockGCOp(7345f51bd6f84b3880838c2460bafb75) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 02:09:44.656862 5639 raft_consensus.cc:998] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5: Attempting to remove follower 4f53d80e39be410d92d4766692aff1c6 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 4f53d80e39be410d92d4766692aff1c6 for more than 300 seconds (300.302s)
I20240502 02:09:44.657210 5639 consensus_queue.cc:237] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:09:44.658834 6917 raft_consensus.cc:1261] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 02:09:44.659348 5687 consensus_queue.cc:1035] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 02:09:44.661845 5639 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.662256 6917 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.665220 6641 catalog_manager.cc:5484] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 reported cstate change: config changed from index 11 to 12, VOTER 4f53d80e39be410d92d4766692aff1c6 (127.27.111.198) evicted. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } }
I20240502 02:09:44.671240 7050 consensus_queue.cc:237] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.672698 6917 raft_consensus.cc:1261] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Refusing update from remote peer c6101231591a458384f4ea6d1f0e0ae5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 02:09:44.673488 5687 consensus_queue.cc:1035] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 02:09:44.673693 6984 consensus_peers.cc:485] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 85e78945910a4e1abfa101d2edc3644e (127.27.111.193:42261): Couldn't send request to peer 85e78945910a4e1abfa101d2edc3644e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14e3c78c65334b52aca79d67c4bf45a2. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.675365 5687 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.675762 6917 raft_consensus.cc:2932] T 14e3c78c65334b52aca79d67c4bf45a2 P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.676936 6630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 14e3c78c65334b52aca79d67c4bf45a2 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:09:44.678138 6637 catalog_manager.cc:5484] T 14e3c78c65334b52aca79d67c4bf45a2 P c6101231591a458384f4ea6d1f0e0ae5 reported cstate change: config changed from index 12 to 13, VOTER 85e78945910a4e1abfa101d2edc3644e (127.27.111.193) added. New cstate: current_term: 1 leader_uuid: "c6101231591a458384f4ea6d1f0e0ae5" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } }
I20240502 02:09:44.749679 5749 raft_consensus.cc:998] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8: Attempting to remove follower 070558c278e0457b9cc070ac0d662b5d from the Raft config. Reason: Leader has been unable to successfully communicate with peer 070558c278e0457b9cc070ac0d662b5d for more than 300 seconds (300.404s)
I20240502 02:09:44.750106 5749 consensus_queue.cc:237] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } }
W20240502 02:09:44.760469 6985 connection.cc:535] client connection to 127.27.111.197:45767 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:09:44.760494 7117 connection.cc:535] client connection to 127.27.111.197:45767 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:09:44.760540 6721 connection.cc:535] client connection to 127.27.111.197:45767 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:09:44.760553 6852 connection.cc:535] client connection to 127.27.111.197:45767 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:09:44.761536 7117 consensus_peers.cc:485] T e6ac26fa6de44f7ca68e6c0b6b959db7 P 86bc8001c3464b929cd5c873861baa0c -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:09:44.761611 7117 consensus_peers.cc:485] T 1c54b912f802461c8cac092ac3222c87 P 86bc8001c3464b929cd5c873861baa0c -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:09:44.761704 6721 consensus_peers.cc:485] T 0359dfbfd11746f1bf62a465f14634ec P 85e78945910a4e1abfa101d2edc3644e -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:09:44.761750 6721 consensus_peers.cc:485] T 4e44267bd266407f9b72c66e3d3579bb P 85e78945910a4e1abfa101d2edc3644e -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:09:44.762213 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111)
W20240502 02:09:44.762279 6985 consensus_peers.cc:485] T 48eeb721a60c4be888041bd31cfeab7a P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:09:44.762315 6985 consensus_peers.cc:462] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111)
W20240502 02:09:44.762293 6852 consensus_peers.cc:485] T 9ecbf1c8ee8c46b195d2374a5db81ac0 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:09:44.762349 6985 consensus_peers.cc:485] T 9313079c48c94ab1b7e022a91befafcd P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:09:44.762413 6985 consensus_peers.cc:485] T a0548274c47148d9b611fd3c978a0c34 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:09:44.779431 6852 consensus_peers.cc:462] T 818dbc9a8e054eeaa122e7919a311806 P e7bcff2fef614e7a9981554c9e6db3c8 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111)
I20240502 02:09:44.806599 5844 ts_tablet_manager.cc:902] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 02:09:44.807103 5844 tablet_copy_client.cc:308] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:36569
I20240502 02:09:44.807597 7201 tablet_copy_service.cc:140] P 86bc8001c3464b929cd5c873861baa0c: Received BeginTabletCopySession request for tablet 8529a93abd6e4d62bbc0bddcf2fda78a from peer e7bcff2fef614e7a9981554c9e6db3c8 ({username='slave'} at 127.27.111.194:34149)
I20240502 02:09:44.807718 7201 tablet_copy_service.cc:161] P 86bc8001c3464b929cd5c873861baa0c: Beginning new tablet copy session on tablet 8529a93abd6e4d62bbc0bddcf2fda78a from peer e7bcff2fef614e7a9981554c9e6db3c8 at {username='slave'} at 127.27.111.194:34149: session id = e7bcff2fef614e7a9981554c9e6db3c8-8529a93abd6e4d62bbc0bddcf2fda78a
I20240502 02:09:44.808918 7201 tablet_copy_source_session.cc:205] T 8529a93abd6e4d62bbc0bddcf2fda78a P 86bc8001c3464b929cd5c873861baa0c: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:09:44.809711 5844 ts_tablet_manager.cc:1875] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:09:44.811646 5844 ts_tablet_manager.cc:1888] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 02:09:44.811805 5844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8529a93abd6e4d62bbc0bddcf2fda78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:44.812651 5844 tablet_copy_client.cc:783] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Starting download of 5 data blocks...
I20240502 02:09:44.814621 5806 ts_tablet_manager.cc:902] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 02:09:44.815128 5806 tablet_copy_client.cc:308] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:36569
I20240502 02:09:44.818436 5844 tablet_copy_client.cc:647] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Starting download of 1 WAL segments...
I20240502 02:09:44.819197 7202 tablet_copy_service.cc:140] P 86bc8001c3464b929cd5c873861baa0c: Received BeginTabletCopySession request for tablet 44c55b199a24432e902cce9121fc0780 from peer 85e78945910a4e1abfa101d2edc3644e ({username='slave'} at 127.27.111.193:33689)
I20240502 02:09:44.819293 7202 tablet_copy_service.cc:161] P 86bc8001c3464b929cd5c873861baa0c: Beginning new tablet copy session on tablet 44c55b199a24432e902cce9121fc0780 from peer 85e78945910a4e1abfa101d2edc3644e at {username='slave'} at 127.27.111.193:33689: session id = 85e78945910a4e1abfa101d2edc3644e-44c55b199a24432e902cce9121fc0780
I20240502 02:09:44.819916 5844 tablet_copy_client.cc:523] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:09:44.820369 7202 tablet_copy_source_session.cc:205] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:09:44.821048 5844 tablet_bootstrap.cc:492] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 02:09:44.821024 5806 ts_tablet_manager.cc:1875] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:09:44.823379 5806 ts_tablet_manager.cc:1888] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 02:09:44.823556 5806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c55b199a24432e902cce9121fc0780. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:44.824630 5806 tablet_copy_client.cc:783] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 5 data blocks...
W20240502 02:09:44.824808 7117 consensus_peers.cc:485] T 44c55b199a24432e902cce9121fc0780 P 86bc8001c3464b929cd5c873861baa0c -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.829146 5806 tablet_copy_client.cc:647] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 1 WAL segments...
I20240502 02:09:44.830855 5844 tablet_bootstrap.cc:492] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:09:44.831069 5806 tablet_copy_client.cc:523] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:09:44.831158 5844 tablet_bootstrap.cc:492] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap complete.
I20240502 02:09:44.831315 5854 ts_tablet_manager.cc:902] T 4154be3f1f3c428580463c419d69892e P e7bcff2fef614e7a9981554c9e6db3c8: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 02:09:44.831328 5844 ts_tablet_manager.cc:1367] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.007s
I20240502 02:09:44.831882 5854 tablet_copy_client.cc:308] T 4154be3f1f3c428580463c419d69892e P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:36569
I20240502 02:09:44.831835 5844 raft_consensus.cc:348] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:09:44.831972 5844 raft_consensus.cc:729] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 02:09:44.832168 5844 consensus_queue.cc:260] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:09:44.832345 7202 tablet_copy_service.cc:140] P 86bc8001c3464b929cd5c873861baa0c: Received BeginTabletCopySession request for tablet 4154be3f1f3c428580463c419d69892e from peer e7bcff2fef614e7a9981554c9e6db3c8 ({username='slave'} at 127.27.111.194:34149)
I20240502 02:09:44.832437 7202 tablet_copy_service.cc:161] P 86bc8001c3464b929cd5c873861baa0c: Beginning new tablet copy session on tablet 4154be3f1f3c428580463c419d69892e from peer e7bcff2fef614e7a9981554c9e6db3c8 at {username='slave'} at 127.27.111.194:34149: session id = e7bcff2fef614e7a9981554c9e6db3c8-4154be3f1f3c428580463c419d69892e
I20240502 02:09:44.832548 5844 ts_tablet_manager.cc:1398] T 8529a93abd6e4d62bbc0bddcf2fda78a P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 02:09:44.832616 5806 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 02:09:44.833494 7202 tablet_copy_source_session.cc:205] T 4154be3f1f3c428580463c419d69892e P 86bc8001c3464b929cd5c873861baa0c: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:09:44.833782 7201 tablet_copy_service.cc:342] P 86bc8001c3464b929cd5c873861baa0c: Request end of tablet copy session e7bcff2fef614e7a9981554c9e6db3c8-8529a93abd6e4d62bbc0bddcf2fda78a received from {username='slave'} at 127.27.111.194:34149
I20240502 02:09:44.833846 7201 tablet_copy_service.cc:433] P 86bc8001c3464b929cd5c873861baa0c: ending tablet copy session e7bcff2fef614e7a9981554c9e6db3c8-8529a93abd6e4d62bbc0bddcf2fda78a on tablet 8529a93abd6e4d62bbc0bddcf2fda78a with peer e7bcff2fef614e7a9981554c9e6db3c8
I20240502 02:09:44.834712 5854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4154be3f1f3c428580463c419d69892e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:44.836745 5854 tablet_copy_client.cc:783] T 4154be3f1f3c428580463c419d69892e P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Starting download of 5 data blocks...
I20240502 02:09:44.840324 5806 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:09:44.840639 5806 tablet_bootstrap.cc:492] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap complete.
I20240502 02:09:44.840793 5806 ts_tablet_manager.cc:1367] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 02:09:44.841142 5806 raft_consensus.cc:348] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.841238 5806 raft_consensus.cc:729] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 02:09:44.841389 5806 consensus_queue.cc:260] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "954cd49326bf405abd81182d75dc783c" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 45767 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.841778 5806 ts_tablet_manager.cc:1398] T 44c55b199a24432e902cce9121fc0780 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 02:09:44.842218 7201 tablet_copy_service.cc:342] P 86bc8001c3464b929cd5c873861baa0c: Request end of tablet copy session 85e78945910a4e1abfa101d2edc3644e-44c55b199a24432e902cce9121fc0780 received from {username='slave'} at 127.27.111.193:33689
I20240502 02:09:44.842305 7201 tablet_copy_service.cc:433] P 86bc8001c3464b929cd5c873861baa0c: ending tablet copy session 85e78945910a4e1abfa101d2edc3644e-44c55b199a24432e902cce9121fc0780 on tablet 44c55b199a24432e902cce9121fc0780 with peer 85e78945910a4e1abfa101d2edc3644e
I20240502 02:09:44.842413 5854 tablet_copy_client.cc:647] T 4154be3f1f3c428580463c419d69892e P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Starting download of 1 WAL segments...
I20240502 02:09:44.843647 5861 ts_tablet_manager.cc:902] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: Initiating tablet copy from peer c6101231591a458384f4ea6d1f0e0ae5 (127.27.111.195:39125)
I20240502 02:09:44.844306 5861 tablet_copy_client.cc:308] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:39125
I20240502 02:09:44.844358 5854 tablet_copy_client.cc:523] T 4154be3f1f3c428580463c419d69892e P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:09:44.844775 7070 tablet_copy_service.cc:140] P c6101231591a458384f4ea6d1f0e0ae5: Received BeginTabletCopySession request for tablet 47a276e9780e448a9b3c6eab67925514 from peer 85e78945910a4e1abfa101d2edc3644e ({username='slave'} at 127.27.111.193:42103)
I20240502 02:09:44.844905 7070 tablet_copy_service.cc:161] P c6101231591a458384f4ea6d1f0e0ae5: Beginning new tablet copy session on tablet 47a276e9780e448a9b3c6eab67925514 from peer 85e78945910a4e1abfa101d2edc3644e at {username='slave'} at 127.27.111.193:42103: session id = 85e78945910a4e1abfa101d2edc3644e-47a276e9780e448a9b3c6eab67925514
I20240502 02:09:44.845692 5854 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 02:09:44.845706 7070 tablet_copy_source_session.cc:205] T 47a276e9780e448a9b3c6eab67925514 P c6101231591a458384f4ea6d1f0e0ae5: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:09:44.846737 5861 ts_tablet_manager.cc:1875] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:09:44.848979 5861 ts_tablet_manager.cc:1888] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 02:09:44.849161 5861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a276e9780e448a9b3c6eab67925514. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:44.850297 5861 tablet_copy_client.cc:783] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 5 data blocks...
I20240502 02:09:44.854414 5854 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:09:44.854738 5854 tablet_bootstrap.cc:492] T 4154be3f1f3c428580463c419d69892e P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap complete.
W20240502 02:09:44.854919 6985 consensus_peers.cc:485] T b69d6e907b3b400f93721981732c006b P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:44.854995 5861 tablet_copy_client.cc:647] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 1 WAL segments...
I20240502 02:09:44.855052 5854 ts_tablet_manager.cc:1367] T 4154be3f1f3c428580463c419d69892e P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.004s
I20240502 02:09:44.855563 5854 raft_consensus.cc:348] T 4154be3f1f3c428580463c419d69892e P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:09:44.855687 5854 raft_consensus.cc:729] T 4154be3f1f3c428580463c419d69892e P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 02:09:44.855844 5854 consensus_queue.cc:260] T 4154be3f1f3c428580463c419d69892e P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:09:44.856302 5861 tablet_copy_client.cc:523] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:09:44.856525 5854 ts_tablet_manager.cc:1398] T 4154be3f1f3c428580463c419d69892e P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:09:44.856998 7201 tablet_copy_service.cc:342] P 86bc8001c3464b929cd5c873861baa0c: Request end of tablet copy session e7bcff2fef614e7a9981554c9e6db3c8-4154be3f1f3c428580463c419d69892e received from {username='slave'} at 127.27.111.194:34149
I20240502 02:09:44.857089 7201 tablet_copy_service.cc:433] P 86bc8001c3464b929cd5c873861baa0c: ending tablet copy session e7bcff2fef614e7a9981554c9e6db3c8-4154be3f1f3c428580463c419d69892e on tablet 4154be3f1f3c428580463c419d69892e with peer e7bcff2fef614e7a9981554c9e6db3c8
I20240502 02:09:44.857781 5861 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 02:09:44.866158 5861 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:09:44.866456 5861 tablet_bootstrap.cc:492] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap complete.
I20240502 02:09:44.866649 5861 ts_tablet_manager.cc:1367] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.009s user 0.010s sys 0.000s
I20240502 02:09:44.867133 5861 raft_consensus.cc:348] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.867234 5861 raft_consensus.cc:729] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 02:09:44.867375 5861 consensus_queue.cc:260] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c6101231591a458384f4ea6d1f0e0ae5" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 39125 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.867650 5861 ts_tablet_manager.cc:1398] T 47a276e9780e448a9b3c6eab67925514 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:09:44.868253 7070 tablet_copy_service.cc:342] P c6101231591a458384f4ea6d1f0e0ae5: Request end of tablet copy session 85e78945910a4e1abfa101d2edc3644e-47a276e9780e448a9b3c6eab67925514 received from {username='slave'} at 127.27.111.193:42103
I20240502 02:09:44.868343 7070 tablet_copy_service.cc:433] P c6101231591a458384f4ea6d1f0e0ae5: ending tablet copy session 85e78945910a4e1abfa101d2edc3644e-47a276e9780e448a9b3c6eab67925514 on tablet 47a276e9780e448a9b3c6eab67925514 with peer 85e78945910a4e1abfa101d2edc3644e
I20240502 02:09:44.887393 5861 ts_tablet_manager.cc:902] T ed7b67af7b1c4255a7c2377570b58e70 P 85e78945910a4e1abfa101d2edc3644e: Initiating tablet copy from peer e7bcff2fef614e7a9981554c9e6db3c8 (127.27.111.194:44305)
I20240502 02:09:44.887921 5861 tablet_copy_client.cc:308] T ed7b67af7b1c4255a7c2377570b58e70 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:44305
I20240502 02:09:44.888393 6937 tablet_copy_service.cc:140] P e7bcff2fef614e7a9981554c9e6db3c8: Received BeginTabletCopySession request for tablet ed7b67af7b1c4255a7c2377570b58e70 from peer 85e78945910a4e1abfa101d2edc3644e ({username='slave'} at 127.27.111.193:40059)
I20240502 02:09:44.888501 6937 tablet_copy_service.cc:161] P e7bcff2fef614e7a9981554c9e6db3c8: Beginning new tablet copy session on tablet ed7b67af7b1c4255a7c2377570b58e70 from peer 85e78945910a4e1abfa101d2edc3644e at {username='slave'} at 127.27.111.193:40059: session id = 85e78945910a4e1abfa101d2edc3644e-ed7b67af7b1c4255a7c2377570b58e70
I20240502 02:09:44.889343 6937 tablet_copy_source_session.cc:205] T ed7b67af7b1c4255a7c2377570b58e70 P e7bcff2fef614e7a9981554c9e6db3c8: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:09:44.890264 5861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed7b67af7b1c4255a7c2377570b58e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:44.890511 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling UndoDeltaBlockGCOp(44c55b199a24432e902cce9121fc0780): 4920 bytes on disk
I20240502 02:09:44.890961 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: UndoDeltaBlockGCOp(44c55b199a24432e902cce9121fc0780) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 02:09:44.891814 6832 maintenance_manager.cc:419] P 85e78945910a4e1abfa101d2edc3644e: Scheduling UndoDeltaBlockGCOp(47a276e9780e448a9b3c6eab67925514): 4581 bytes on disk
I20240502 02:09:44.892204 6722 maintenance_manager.cc:643] P 85e78945910a4e1abfa101d2edc3644e: UndoDeltaBlockGCOp(47a276e9780e448a9b3c6eab67925514) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240502 02:09:44.892838 5861 tablet_copy_client.cc:783] T ed7b67af7b1c4255a7c2377570b58e70 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 5 data blocks...
I20240502 02:09:44.896950 5861 tablet_copy_client.cc:647] T ed7b67af7b1c4255a7c2377570b58e70 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Starting download of 1 WAL segments...
I20240502 02:09:44.898180 5861 tablet_copy_client.cc:523] T ed7b67af7b1c4255a7c2377570b58e70 P 85e78945910a4e1abfa101d2edc3644e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:09:44.899495 5861 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap starting.
I20240502 02:09:44.907227 5861 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:09:44.907490 5861 tablet_bootstrap.cc:492] T ed7b67af7b1c4255a7c2377570b58e70 P 85e78945910a4e1abfa101d2edc3644e: Bootstrap complete.
I20240502 02:09:44.907653 5861 ts_tablet_manager.cc:1367] T ed7b67af7b1c4255a7c2377570b58e70 P 85e78945910a4e1abfa101d2edc3644e: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20240502 02:09:44.908008 5861 raft_consensus.cc:348] T ed7b67af7b1c4255a7c2377570b58e70 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.908113 5861 raft_consensus.cc:729] T ed7b67af7b1c4255a7c2377570b58e70 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e78945910a4e1abfa101d2edc3644e, State: Initialized, Role: FOLLOWER
I20240502 02:09:44.908234 5861 consensus_queue.cc:260] T ed7b67af7b1c4255a7c2377570b58e70 P 85e78945910a4e1abfa101d2edc3644e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } } peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } }
I20240502 02:09:44.908474 5861 ts_tablet_manager.cc:1398] T ed7b67af7b1c4255a7c2377570b58e70 P 85e78945910a4e1abfa101d2edc3644e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:09:44.908958 6937 tablet_copy_service.cc:342] P e7bcff2fef614e7a9981554c9e6db3c8: Request end of tablet copy session 85e78945910a4e1abfa101d2edc3644e-ed7b67af7b1c4255a7c2377570b58e70 received from {username='slave'} at 127.27.111.193:40059
I20240502 02:09:44.909046 6937 tablet_copy_service.cc:433] P e7bcff2fef614e7a9981554c9e6db3c8: ending tablet copy session 85e78945910a4e1abfa101d2edc3644e-ed7b67af7b1c4255a7c2377570b58e70 on tablet ed7b67af7b1c4255a7c2377570b58e70 with peer 85e78945910a4e1abfa101d2edc3644e
I20240502 02:09:44.918921 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling UndoDeltaBlockGCOp(8529a93abd6e4d62bbc0bddcf2fda78a): 4834 bytes on disk
I20240502 02:09:44.919359 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: UndoDeltaBlockGCOp(8529a93abd6e4d62bbc0bddcf2fda78a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 02:09:44.919942 6964 maintenance_manager.cc:419] P e7bcff2fef614e7a9981554c9e6db3c8: Scheduling UndoDeltaBlockGCOp(4154be3f1f3c428580463c419d69892e): 4581 bytes on disk
I20240502 02:09:44.920229 6853 maintenance_manager.cc:643] P e7bcff2fef614e7a9981554c9e6db3c8: UndoDeltaBlockGCOp(4154be3f1f3c428580463c419d69892e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4}
W20240502 02:09:44.931484 6985 consensus_peers.cc:485] T 7345f51bd6f84b3880838c2460bafb75 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:09:44.952877 6985 consensus_peers.cc:462] T e0e05f96dbc542eb999c16dcc0620666 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111)
I20240502 02:09:44.961575 5854 ts_tablet_manager.cc:902] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: Initiating tablet copy from peer 86bc8001c3464b929cd5c873861baa0c (127.27.111.196:36569)
I20240502 02:09:44.962136 5854 tablet_copy_client.cc:308] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.196:36569
I20240502 02:09:44.962649 7201 tablet_copy_service.cc:140] P 86bc8001c3464b929cd5c873861baa0c: Received BeginTabletCopySession request for tablet 7fa358140eb742b587a4d6a203738b4f from peer e7bcff2fef614e7a9981554c9e6db3c8 ({username='slave'} at 127.27.111.194:34149)
I20240502 02:09:44.962755 7201 tablet_copy_service.cc:161] P 86bc8001c3464b929cd5c873861baa0c: Beginning new tablet copy session on tablet 7fa358140eb742b587a4d6a203738b4f from peer e7bcff2fef614e7a9981554c9e6db3c8 at {username='slave'} at 127.27.111.194:34149: session id = e7bcff2fef614e7a9981554c9e6db3c8-7fa358140eb742b587a4d6a203738b4f
I20240502 02:09:44.963646 7201 tablet_copy_source_session.cc:205] T 7fa358140eb742b587a4d6a203738b4f P 86bc8001c3464b929cd5c873861baa0c: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:09:44.964368 5854 ts_tablet_manager.cc:1875] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:09:44.966681 5854 ts_tablet_manager.cc:1888] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 02:09:44.966852 5854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fa358140eb742b587a4d6a203738b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:44.968205 5854 tablet_copy_client.cc:783] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Starting download of 5 data blocks...
W20240502 02:09:44.969600 6985 consensus_peers.cc:462] T 91db5c1b3d2f4d22b5882a61ea0395c7 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111)
I20240502 02:09:44.973636 5854 tablet_copy_client.cc:647] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Starting download of 1 WAL segments...
I20240502 02:09:44.975169 5854 tablet_copy_client.cc:523] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:09:44.976387 5854 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap starting.
I20240502 02:09:44.983681 5854 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:09:44.984066 5854 tablet_bootstrap.cc:492] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: Bootstrap complete.
I20240502 02:09:44.984246 5854 ts_tablet_manager.cc:1367] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240502 02:09:44.984652 5854 raft_consensus.cc:348] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:09:44.984771 5854 raft_consensus.cc:729] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bcff2fef614e7a9981554c9e6db3c8, State: Initialized, Role: FOLLOWER
I20240502 02:09:44.984901 5854 consensus_queue.cc:260] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "86bc8001c3464b929cd5c873861baa0c" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 36569 } } peers { permanent_uuid: "85e78945910a4e1abfa101d2edc3644e" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 42261 } } peers { permanent_uuid: "e7bcff2fef614e7a9981554c9e6db3c8" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 44305 } }
I20240502 02:09:44.985253 5854 ts_tablet_manager.cc:1398] T 7fa358140eb742b587a4d6a203738b4f P e7bcff2fef614e7a9981554c9e6db3c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20240502 02:09:44.990808 6718 consensus_peers.cc:462] T a4fb9af11abc49d496b5c3d8048c22c8 P 85e78945910a4e1abfa101d2edc3644e -> Peer 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:44.992792 7201 tablet_copy_service.cc:342] P 86bc8001c3464b929cd5c873861baa0c: Request end of tablet copy session e7bcff2fef614e7a9981554c9e6db3c8-7fa358140eb742b587a4d6a203738b4f received from {username='slave'} at 127.27.111.194:34149
I20240502 02:09:44.992877 7201 tablet_copy_service.cc:433] P 86bc8001c3464b929cd5c873861baa0c: ending tablet copy session e7bcff2fef614e7a9981554c9e6db3c8-7fa358140eb742b587a4d6a203738b4f on tablet 7fa358140eb742b587a4d6a203738b4f with peer e7bcff2fef614e7a9981554c9e6db3c8
I20240502 02:09:45.013054 6783 raft_consensus.cc:1203] T 7345f51bd6f84b3880838c2460bafb75 P 85e78945910a4e1abfa101d2edc3644e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 02:09:45.044970 7161 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:09:45.045214 6763 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:09:45.048444 7030 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 02:09:45.053135 6894 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:09:45.053619 6627 catalog_manager.cc:4573] TS 070558c278e0457b9cc070ac0d662b5d (127.27.111.200:37235): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4154be3f1f3c428580463c419d69892e: Network error: Client connection negotiation failed: client connection to 127.27.111.200:37235: connect: Connection refused (error 111)
I20240502 02:09:45.089828 8953 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task43FDSt/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 01:59:24.520511 8958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:59:24.520913 8958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:59:24.525489 8958 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:59:25.650444 8958 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:59:26.180974 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.181124 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.181213 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.181272 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.181309 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.181416 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.181460 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.181532 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.181574 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.181603 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.181738 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.181778 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.181813 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.181860 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
I20240502 01:59:26.275321 8958 rebalancer_tool.cc:1074] tablet 17235e019e644b258266636fb6be6648: '86bc8001c3464b929cd5c873861baa0c' -> '4f53d80e39be410d92d4766692aff1c6' move scheduled
I20240502 01:59:26.324486 8958 rebalancer_tool.cc:1084] tablet ed7b67af7b1c4255a7c2377570b58e70: 'c6101231591a458384f4ea6d1f0e0ae5' -> '070558c278e0457b9cc070ac0d662b5d' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 01:59:26.324616 8958 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:26.624346 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.624533 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.624622 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.624660 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.624734 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.624882 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.624939 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.624977 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.625039 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.625072 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.625101 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.625146 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.625193 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.625237 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.625267 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.625291 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.625324 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:26.625352 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:26.625383 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:26.625417 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:26.625455 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:26.706180 8958 rebalancer_tool.cc:1074] tablet 9f6c07e3923d47f29ef32561d7f438e1: 'c6101231591a458384f4ea6d1f0e0ae5' -> '070558c278e0457b9cc070ac0d662b5d' move scheduled
I20240502 01:59:26.827924 8958 rebalancer_tool.cc:1084] tablet 8529a93abd6e4d62bbc0bddcf2fda78a: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move ignored: Illegal state: tablet 8529a93abd6e4d62bbc0bddcf2fda78a: replica 4e6e76b381af4cdc90bd3e9e6e8b5cf2 is already present
I20240502 01:59:26.828030 8958 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:27.587278 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.587422 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.587530 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.587638 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.587735 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.587800 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.587857 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.587942 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.587999 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9d0e8c4a95a747b18b0b6422ed894715: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.588044 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:27.588089 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.588143 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:27.588212 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:27.588268 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:27.588352 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
I20240502 01:59:27.649259 8958 rebalancer_tool.cc:1074] tablet a0548274c47148d9b611fd3c978a0c34: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:27.722445 8958 rebalancer_tool.cc:1074] tablet c8c237c50773494abf8813494be0f184: '85e78945910a4e1abfa101d2edc3644e' -> '4f53d80e39be410d92d4766692aff1c6' move scheduled
I20240502 01:59:27.852947 8958 rebalancer_tool.cc:1074] tablet 4e44267bd266407f9b72c66e3d3579bb: '86bc8001c3464b929cd5c873861baa0c' -> '954cd49326bf405abd81182d75dc783c' move scheduled
I20240502 01:59:27.894227 8958 rebalancer_tool.cc:1084] tablet e0e05f96dbc542eb999c16dcc0620666: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move ignored: Illegal state: tablet e0e05f96dbc542eb999c16dcc0620666: replica 4e6e76b381af4cdc90bd3e9e6e8b5cf2 is already present
I20240502 01:59:27.894318 8958 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:28.426927 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.427013 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:28.427065 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.427129 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:28.427167 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.427213 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.427268 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:28.427321 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:28.427381 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f2b695d4b3e54fdfad81f019e9ab7b4a: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.427413 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:28.427443 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:28.427507 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:28.427536 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 954cd49326bf405abd81182d75dc783c
W20240502 01:59:28.427565 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:28.598595 8958 rebalancer_tool.cc:1074] tablet dcc5daef41bc44c4855f2bcb44e1c6d5: '86bc8001c3464b929cd5c873861baa0c' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move scheduled
I20240502 01:59:28.637467 8958 rebalancer_tool.cc:1074] tablet 347b161e9ec84640a0668836ce9e29eb: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:28.647689 8958 rebalancer_tool.cc:1084] tablet b69d6e907b3b400f93721981732c006b: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '954cd49326bf405abd81182d75dc783c' move ignored: Illegal state: tablet b69d6e907b3b400f93721981732c006b: replica 954cd49326bf405abd81182d75dc783c is already present
I20240502 01:59:28.647784 8958 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:59:29.084406 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 19f17ecb48ac4307bcd5af6811f21abe: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:29.084595 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:29.084662 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6397b955acf741aeb8f869ab987d74b0: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:29.084725 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3f6f56d5459f45faa8baa3e7d3fb95cb: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:29.084774 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ed198b9f6450449ca6bd353bdb977c91: could not find any suitable replica to move from server 86bc8001c3464b929cd5c873861baa0c to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:29.084823 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 331b273e69914efb86a2391b30c6d6cf: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4f53d80e39be410d92d4766692aff1c6
W20240502 01:59:29.084887 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table afb5b6283779407b9ddcb26793f68cf5: could not find any suitable replica to move from server c6101231591a458384f4ea6d1f0e0ae5 to server 070558c278e0457b9cc070ac0d662b5d
W20240502 01:59:29.084941 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table bfc1490f5eee4d0bb5455e4b3ee959f1: could not find any suitable replica to move from server 85e78945910a4e1abfa101d2edc3644e to server 4e6e76b381af4cdc90bd3e9e6e8b5cf2
W20240502 01:59:29.084990 8958 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eda9a0cbf0634fa49df0814b3e1ce66c: could not find any suitable replica to move from server e7bcff2fef614e7a9981554c9e6db3c8 to server 4f53d80e39be410d92d4766692aff1c6
I20240502 01:59:29.097088 8958 rebalancer_tool.cc:1074] tablet 9313079c48c94ab1b7e022a91befafcd: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:29.106885 8958 rebalancer_tool.cc:1074] tablet 0359dfbfd11746f1bf62a465f14634ec: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:29.146696 8958 rebalancer_tool.cc:1074] tablet 4154be3f1f3c428580463c419d69892e: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:29.975066 8958 rebalancer_tool.cc:1136] tablet 9313079c48c94ab1b7e022a91befafcd: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:31.655678 8958 rebalancer_tool.cc:1136] tablet 9f6c07e3923d47f29ef32561d7f438e1: 'c6101231591a458384f4ea6d1f0e0ae5' -> '070558c278e0457b9cc070ac0d662b5d' move completed: OK
I20240502 01:59:32.282042 8958 rebalancer_tool.cc:1136] tablet 17235e019e644b258266636fb6be6648: '86bc8001c3464b929cd5c873861baa0c' -> '4f53d80e39be410d92d4766692aff1c6' move completed: OK
I20240502 01:59:32.293906 8958 rebalancer_tool.cc:1136] tablet a0548274c47148d9b611fd3c978a0c34: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:33.359692 8958 rebalancer_tool.cc:1074] tablet 8529a93abd6e4d62bbc0bddcf2fda78a: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:33.383738 8958 rebalancer_tool.cc:1074] tablet e6ac26fa6de44f7ca68e6c0b6b959db7: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:33.893110 8958 rebalancer_tool.cc:1136] tablet 4154be3f1f3c428580463c419d69892e: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:33.904001 8958 rebalancer_tool.cc:1136] tablet 0359dfbfd11746f1bf62a465f14634ec: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:33.914759 8958 rebalancer_tool.cc:1136] tablet 347b161e9ec84640a0668836ce9e29eb: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:33.922571 8958 rebalancer_tool.cc:1136] tablet 8529a93abd6e4d62bbc0bddcf2fda78a: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:33.931735 8958 rebalancer_tool.cc:1136] tablet 4e44267bd266407f9b72c66e3d3579bb: '86bc8001c3464b929cd5c873861baa0c' -> '954cd49326bf405abd81182d75dc783c' move completed: OK
I20240502 01:59:33.952105 8958 rebalancer_tool.cc:1136] tablet dcc5daef41bc44c4855f2bcb44e1c6d5: '86bc8001c3464b929cd5c873861baa0c' -> '4e6e76b381af4cdc90bd3e9e6e8b5cf2' move completed: OK
I20240502 01:59:34.539115 8958 rebalancer_tool.cc:1074] tablet a66b932d884e4df8a78841103d88ded0: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:34.552682 8958 rebalancer_tool.cc:1074] tablet 7345f51bd6f84b3880838c2460bafb75: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:34.562523 8958 rebalancer_tool.cc:1074] tablet 034cf1c3d47a4f7785c16d16815c229d: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:34.575304 8958 rebalancer_tool.cc:1074] tablet 9ecbf1c8ee8c46b195d2374a5db81ac0: '85e78945910a4e1abfa101d2edc3644e' -> '' move scheduled
I20240502 01:59:34.585016 8958 rebalancer_tool.cc:1136] tablet 9ecbf1c8ee8c46b195d2374a5db81ac0: '85e78945910a4e1abfa101d2edc3644e' -> '' move completed: OK
I20240502 01:59:35.453631 8958 rebalancer_tool.cc:1136] tablet 7345f51bd6f84b3880838c2460bafb75: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:35.470528 8958 rebalancer_tool.cc:1136] tablet a66b932d884e4df8a78841103d88ded0: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:35.479775 8958 rebalancer_tool.cc:1136] tablet e6ac26fa6de44f7ca68e6c0b6b959db7: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:35.498313 8958 rebalancer_tool.cc:1136] tablet c8c237c50773494abf8813494be0f184: '85e78945910a4e1abfa101d2edc3644e' -> '4f53d80e39be410d92d4766692aff1c6' move completed: OK
I20240502 01:59:35.509096 8958 rebalancer_tool.cc:1074] tablet 47a276e9780e448a9b3c6eab67925514: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:35.529505 8958 rebalancer_tool.cc:1074] tablet e0e05f96dbc542eb999c16dcc0620666: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move scheduled
I20240502 01:59:35.628458 8958 rebalancer_tool.cc:1074] tablet 818dbc9a8e054eeaa122e7919a311806: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move scheduled
I20240502 01:59:35.663193 8958 rebalancer_tool.cc:1074] tablet 39d9e48f67f34ebe850fc388ecd94ca3: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:35.691267 8958 rebalancer_tool.cc:1136] tablet 818dbc9a8e054eeaa122e7919a311806: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:35.719549 8958 rebalancer_tool.cc:1136] tablet e0e05f96dbc542eb999c16dcc0620666: 'e7bcff2fef614e7a9981554c9e6db3c8' -> '' move completed: OK
I20240502 01:59:36.303066 8958 rebalancer_tool.cc:1136] tablet 39d9e48f67f34ebe850fc388ecd94ca3: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:36.311249 8958 rebalancer_tool.cc:1136] tablet 034cf1c3d47a4f7785c16d16815c229d: 'c6101231591a458384f4ea6d1f0e0ae5' -> '' move completed: OK
I20240502 01:59:36.326076 8958 rebalancer_tool.cc:1074] tablet 91db5c1b3d2f4d22b5882a61ea0395c7: '86bc8001c3464b929cd5c873861baa0c' -> '' move scheduled
I20240502 01:59:37.674801 8958 rebalancer_tool.cc:1136] tablet 47a276e9780e448a9b3c6eab67925514: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:37.771353 8958 rebalancer_tool.cc:1136] tablet 91db5c1b3d2f4d22b5882a61ea0395c7: '86bc8001c3464b929cd5c873861baa0c' -> '' move completed: OK
I20240502 01:59:38.354470 8958 rebalancer_tool.cc:1074] tablet
E20240502 02:09:45.090317 28095 external_mini_cluster.cc:943] Process with UUID 954cd49326bf405abd81182d75dc783c has crashed
E20240502 02:09:45.090447 28095 external_mini_cluster.cc:943] Process with UUID 4f53d80e39be410d92d4766692aff1c6 has crashed
E20240502 02:09:45.090512 28095 external_mini_cluster.cc:943] Process with UUID 4e6e76b381af4cdc90bd3e9e6e8b5cf2 has crashed
E20240502 02:09:45.090570 28095 external_mini_cluster.cc:943] Process with UUID 070558c278e0457b9cc070ac0d662b5d has crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 4
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240502 02:09:45.091346 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 6703
I20240502 02:09:45.100315 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 6834
W20240502 02:09:45.101918 6985 consensus_peers.cc:485] T d4d429fe60e74108bc0c9ca8509365f2 P c6101231591a458384f4ea6d1f0e0ae5 -> Peer 954cd49326bf405abd81182d75dc783c (127.27.111.197:45767): Couldn't send request to peer 954cd49326bf405abd81182d75dc783c. Status: Network error: Client connection negotiation failed: client connection to 127.27.111.197:45767: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:45.108912 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 6966
I20240502 02:09:45.117308 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 7099
I20240502 02:09:45.125674 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 6612
2024-05-02T02:09:45Z chronyd exiting
I20240502 02:09:45.144860 28095 test_util.cc:175] -----------------------------------------------
I20240502 02:09:45.144924 28095 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714615075754784-28095-0
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00> (632587 ms)
[----------] 1 test from ConcurrentRebalancersTest (632587 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0
I20240502 02:09:45.145704 28095 test_util.cc:268] Using random seed: 416073061
I20240502 02:09:45.146589 28095 ts_itest-base.cc:112] Starting cluster with:
I20240502 02:09:45.146631 28095 ts_itest-base.cc:113] --------------
I20240502 02:09:45.146651 28095 ts_itest-base.cc:114] 4 tablet servers
I20240502 02:09:45.146667 28095 ts_itest-base.cc:115] 1 replicas per TS
I20240502 02:09:45.146695 28095 ts_itest-base.cc:116] --------------
2024-05-02T02:09:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T02:09:45Z Disabled control of system clock
I20240502 02:09:45.155270 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.111.254:36011
--webserver_interface=127.27.111.254
--webserver_port=0
--builtin_ntp_servers=127.27.111.212:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.111.254:36011
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240502 02:09:45.266120 5927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:45.266355 5927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:45.266438 5927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:45.269470 5927 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 02:09:45.269541 5927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 02:09:45.269563 5927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:45.269580 5927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 02:09:45.269596 5927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 02:09:45.273090 5927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.27.111.254:36011
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.111.254:36011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.111.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.5927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:45.274024 5927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:45.275174 5927 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:45.282743 5933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:45.283262 5932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:45.283991 5935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:45.284571 5927 server_base.cc:1034] running on GCE node
I20240502 02:09:45.285017 5927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:45.285926 5927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:45.287075 5927 hybrid_clock.cc:648] HybridClock initialized: now 1714615785287068 us; error 26 us; skew 500 ppm
I20240502 02:09:45.288911 5927 webserver.cc:460] Webserver started at http://127.27.111.254:43405/ using document root <none> and password file <none>
I20240502 02:09:45.289461 5927 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:45.289516 5927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:45.289695 5927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:09:45.291370 5927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1c62d235e6c14ce1ba75889238494eaa"
format_stamp: "Formatted at 2024-05-02 02:09:45 on dist-test-slave-0pw1"
I20240502 02:09:45.291853 5927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1c62d235e6c14ce1ba75889238494eaa"
format_stamp: "Formatted at 2024-05-02 02:09:45 on dist-test-slave-0pw1"
I20240502 02:09:45.295284 5927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 02:09:45.297430 5942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:45.298444 5927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 02:09:45.298553 5927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1c62d235e6c14ce1ba75889238494eaa"
format_stamp: "Formatted at 2024-05-02 02:09:45 on dist-test-slave-0pw1"
I20240502 02:09:45.298647 5927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:45.326156 5927 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:45.326894 5927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:45.327065 5927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:45.333853 5927 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.254:36011
I20240502 02:09:45.333863 5995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.254:36011 every 8 connection(s)
I20240502 02:09:45.334883 5927 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 02:09:45.337411 5996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:45.341529 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 5927
I20240502 02:09:45.341647 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 02:09:45.342592 5996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa: Bootstrap starting.
I20240502 02:09:45.344694 5996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:45.345535 5996 log.cc:824] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa: Log is configured to *not* fsync() on all Append() calls
I20240502 02:09:45.347404 5996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa: No bootstrap required, opened a new log
I20240502 02:09:45.350226 5996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c62d235e6c14ce1ba75889238494eaa" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 36011 } }
I20240502 02:09:45.350462 5996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:45.350528 5996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c62d235e6c14ce1ba75889238494eaa, State: Initialized, Role: FOLLOWER
I20240502 02:09:45.350955 5996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c62d235e6c14ce1ba75889238494eaa" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 36011 } }
I20240502 02:09:45.351094 5996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:45.351153 5996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:45.351272 5996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:45.352039 5996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c62d235e6c14ce1ba75889238494eaa" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 36011 } }
I20240502 02:09:45.352317 5996 leader_election.cc:302] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c62d235e6c14ce1ba75889238494eaa; no voters:
I20240502 02:09:45.352553 5996 leader_election.cc:288] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:45.352891 6001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:45.353081 6001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [term 1 LEADER]: Becoming Leader. State: Replica: 1c62d235e6c14ce1ba75889238494eaa, State: Running, Role: LEADER
I20240502 02:09:45.353339 6001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c62d235e6c14ce1ba75889238494eaa" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 36011 } }
I20240502 02:09:45.353691 5996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [sys.catalog]: configured and running, proceeding with master startup.
I20240502 02:09:45.354818 6002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c62d235e6c14ce1ba75889238494eaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c62d235e6c14ce1ba75889238494eaa" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 36011 } } }
I20240502 02:09:45.354938 6002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [sys.catalog]: This master's current role is: LEADER
I20240502 02:09:45.355229 6003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c62d235e6c14ce1ba75889238494eaa. Latest consensus state: current_term: 1 leader_uuid: "1c62d235e6c14ce1ba75889238494eaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c62d235e6c14ce1ba75889238494eaa" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 36011 } } }
I20240502 02:09:45.355314 6003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa [sys.catalog]: This master's current role is: LEADER
I20240502 02:09:45.355484 6010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 02:09:45.358233 6010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 02:09:45.363425 6010 catalog_manager.cc:1347] Generated new cluster ID: 8f0f21a4d1d04a069e0e3c131a4a7a21
I20240502 02:09:45.363504 6010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 02:09:45.373811 6010 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 02:09:45.375105 6010 catalog_manager.cc:1504] Loading token signing keys...
I20240502 02:09:45.381877 6010 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1c62d235e6c14ce1ba75889238494eaa: Generated new TSK 0
I20240502 02:09:45.382774 6010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 02:09:45.391314 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.193:0
--local_ip_for_outbound_sockets=127.27.111.193
--webserver_interface=127.27.111.193
--webserver_port=0
--tserver_master_addrs=127.27.111.254:36011
--builtin_ntp_servers=127.27.111.212:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 02:09:45.503072 6020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:45.503360 6020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:45.503425 6020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:45.506534 6020 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 02:09:45.506613 6020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:45.506711 6020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.193
I20240502 02:09:45.510309 6020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.111.193
--webserver_port=0
--tserver_master_addrs=127.27.111.254:36011
--never_fsync=true
--heap_profile_path=/tmp/kudu.6020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.193
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:45.511276 6020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:45.512385 6020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:45.520915 6028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:45.521562 6025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:45.522150 6026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:45.522935 6020 server_base.cc:1034] running on GCE node
I20240502 02:09:45.523345 6020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:45.524004 6020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:45.525163 6020 hybrid_clock.cc:648] HybridClock initialized: now 1714615785525151 us; error 35 us; skew 500 ppm
I20240502 02:09:45.527144 6020 webserver.cc:460] Webserver started at http://127.27.111.193:45829/ using document root <none> and password file <none>
I20240502 02:09:45.527683 6020 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:45.527735 6020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:45.527901 6020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:09:45.529600 6020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "05f91f1eaf8349178678a275be055207"
format_stamp: "Formatted at 2024-05-02 02:09:45 on dist-test-slave-0pw1"
I20240502 02:09:45.530061 6020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "05f91f1eaf8349178678a275be055207"
format_stamp: "Formatted at 2024-05-02 02:09:45 on dist-test-slave-0pw1"
I20240502 02:09:45.533556 6020 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 02:09:45.535836 6034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:45.536917 6020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 02:09:45.537024 6020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "05f91f1eaf8349178678a275be055207"
format_stamp: "Formatted at 2024-05-02 02:09:45 on dist-test-slave-0pw1"
I20240502 02:09:45.537118 6020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:45.558799 6020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:45.559449 6020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:45.559597 6020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:45.560127 6020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:09:45.561009 6020 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:09:45.561066 6020 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:45.561110 6020 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:09:45.561131 6020 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:45.570001 6020 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.193:44903
I20240502 02:09:45.570036 6147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.193:44903 every 8 connection(s)
I20240502 02:09:45.571012 6020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 02:09:45.577693 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 6020
I20240502 02:09:45.577814 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 02:09:45.579948 6148 heartbeater.cc:347] Connected to a master server at 127.27.111.254:36011
I20240502 02:09:45.580185 6148 heartbeater.cc:464] Registering TS with master...
I20240502 02:09:45.580530 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.194:0
--local_ip_for_outbound_sockets=127.27.111.194
--webserver_interface=127.27.111.194
--webserver_port=0
--tserver_master_addrs=127.27.111.254:36011
--builtin_ntp_servers=127.27.111.212:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 02:09:45.580610 6148 heartbeater.cc:511] Master 127.27.111.254:36011 requested a full tablet report, sending...
I20240502 02:09:45.581940 5958 ts_manager.cc:194] Registered new tserver with Master: 05f91f1eaf8349178678a275be055207 (127.27.111.193:44903)
I20240502 02:09:45.583158 5958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.193:47033
W20240502 02:09:45.695333 6151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:45.695559 6151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:45.695608 6151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:45.698643 6151 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 02:09:45.698714 6151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:45.698797 6151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.194
I20240502 02:09:45.702370 6151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.111.194
--webserver_port=0
--tserver_master_addrs=127.27.111.254:36011
--never_fsync=true
--heap_profile_path=/tmp/kudu.6151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.194
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:45.703297 6151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:45.704412 6151 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:45.712908 6156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:45.713470 6159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:45.714270 6157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:45.714879 6151 server_base.cc:1034] running on GCE node
I20240502 02:09:45.715302 6151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:45.715965 6151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:45.717116 6151 hybrid_clock.cc:648] HybridClock initialized: now 1714615785717108 us; error 28 us; skew 500 ppm
I20240502 02:09:45.718995 6151 webserver.cc:460] Webserver started at http://127.27.111.194:39815/ using document root <none> and password file <none>
I20240502 02:09:45.719552 6151 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:45.719616 6151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:45.719790 6151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:09:45.721410 6151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f659a9953c2341c48931e531cc59c781"
format_stamp: "Formatted at 2024-05-02 02:09:45 on dist-test-slave-0pw1"
I20240502 02:09:45.721880 6151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f659a9953c2341c48931e531cc59c781"
format_stamp: "Formatted at 2024-05-02 02:09:45 on dist-test-slave-0pw1"
I20240502 02:09:45.725106 6151 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 02:09:45.727205 6165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:45.728183 6151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:45.728291 6151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f659a9953c2341c48931e531cc59c781"
format_stamp: "Formatted at 2024-05-02 02:09:45 on dist-test-slave-0pw1"
I20240502 02:09:45.728371 6151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:45.742467 6151 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:45.743093 6151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:45.743255 6151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:45.743799 6151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:09:45.744688 6151 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:09:45.744740 6151 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:45.744800 6151 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:09:45.744827 6151 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:45.753118 6151 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.194:33833
I20240502 02:09:45.753140 6279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.194:33833 every 8 connection(s)
I20240502 02:09:45.754083 6151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 02:09:45.756508 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 6151
I20240502 02:09:45.756644 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 02:09:45.758829 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.195:0
--local_ip_for_outbound_sockets=127.27.111.195
--webserver_interface=127.27.111.195
--webserver_port=0
--tserver_master_addrs=127.27.111.254:36011
--builtin_ntp_servers=127.27.111.212:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 02:09:45.763118 6280 heartbeater.cc:347] Connected to a master server at 127.27.111.254:36011
I20240502 02:09:45.763387 6280 heartbeater.cc:464] Registering TS with master...
I20240502 02:09:45.763989 6280 heartbeater.cc:511] Master 127.27.111.254:36011 requested a full tablet report, sending...
I20240502 02:09:45.765064 5958 ts_manager.cc:194] Registered new tserver with Master: f659a9953c2341c48931e531cc59c781 (127.27.111.194:33833)
I20240502 02:09:45.765614 5958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.194:37205
W20240502 02:09:45.867067 6283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:45.867290 6283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:45.867338 6283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:45.870465 6283 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 02:09:45.870532 6283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:45.870625 6283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.195
I20240502 02:09:45.874197 6283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.111.195
--webserver_port=0
--tserver_master_addrs=127.27.111.254:36011
--never_fsync=true
--heap_profile_path=/tmp/kudu.6283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.195
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:45.875118 6283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:45.876242 6283 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:45.884168 6288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:45.884680 6289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:45.885345 6291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:45.885963 6283 server_base.cc:1034] running on GCE node
I20240502 02:09:45.886318 6283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:45.886948 6283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:45.888098 6283 hybrid_clock.cc:648] HybridClock initialized: now 1714615785888086 us; error 27 us; skew 500 ppm
I20240502 02:09:45.889880 6283 webserver.cc:460] Webserver started at http://127.27.111.195:36125/ using document root <none> and password file <none>
I20240502 02:09:45.890473 6283 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:45.890543 6283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:45.890728 6283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:09:45.892354 6283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4fe3bba8e9f14ba79a92f12af08feaca"
format_stamp: "Formatted at 2024-05-02 02:09:45 on dist-test-slave-0pw1"
I20240502 02:09:45.892812 6283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4fe3bba8e9f14ba79a92f12af08feaca"
format_stamp: "Formatted at 2024-05-02 02:09:45 on dist-test-slave-0pw1"
I20240502 02:09:45.896091 6283 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 02:09:45.898092 6297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:45.899127 6283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:45.899237 6283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4fe3bba8e9f14ba79a92f12af08feaca"
format_stamp: "Formatted at 2024-05-02 02:09:45 on dist-test-slave-0pw1"
I20240502 02:09:45.899322 6283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:45.909217 6283 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:45.909775 6283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:45.909927 6283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:45.910456 6283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:09:45.911289 6283 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:09:45.911338 6283 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:45.911396 6283 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:09:45.911427 6283 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:45.919414 6283 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.195:37645
I20240502 02:09:45.919445 6412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.195:37645 every 8 connection(s)
I20240502 02:09:45.920329 6283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 02:09:45.924523 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 6283
I20240502 02:09:45.924623 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 02:09:45.926656 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.196:0
--local_ip_for_outbound_sockets=127.27.111.196
--webserver_interface=127.27.111.196
--webserver_port=0
--tserver_master_addrs=127.27.111.254:36011
--builtin_ntp_servers=127.27.111.212:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 02:09:45.928810 6413 heartbeater.cc:347] Connected to a master server at 127.27.111.254:36011
I20240502 02:09:45.929049 6413 heartbeater.cc:464] Registering TS with master...
I20240502 02:09:45.929598 6413 heartbeater.cc:511] Master 127.27.111.254:36011 requested a full tablet report, sending...
I20240502 02:09:45.930686 5958 ts_manager.cc:194] Registered new tserver with Master: 4fe3bba8e9f14ba79a92f12af08feaca (127.27.111.195:37645)
I20240502 02:09:45.931241 5958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.195:57591
W20240502 02:09:46.034862 6416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:46.035073 6416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:46.035125 6416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:46.038144 6416 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 02:09:46.038218 6416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:46.038286 6416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.196
I20240502 02:09:46.041909 6416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.111.196
--webserver_port=0
--tserver_master_addrs=127.27.111.254:36011
--never_fsync=true
--heap_profile_path=/tmp/kudu.6416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.196
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:46.042831 6416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:46.043936 6416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:46.052480 6422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:46.053022 6421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:46.053733 6424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:46.054358 6416 server_base.cc:1034] running on GCE node
I20240502 02:09:46.054764 6416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:46.055444 6416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:46.056607 6416 hybrid_clock.cc:648] HybridClock initialized: now 1714615786056597 us; error 23 us; skew 500 ppm
I20240502 02:09:46.058316 6416 webserver.cc:460] Webserver started at http://127.27.111.196:45045/ using document root <none> and password file <none>
I20240502 02:09:46.058863 6416 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:46.058923 6416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:46.059108 6416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:09:46.060734 6416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "dc53186baf544ae4a4a92df94b0a3607"
format_stamp: "Formatted at 2024-05-02 02:09:46 on dist-test-slave-0pw1"
I20240502 02:09:46.061198 6416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "dc53186baf544ae4a4a92df94b0a3607"
format_stamp: "Formatted at 2024-05-02 02:09:46 on dist-test-slave-0pw1"
I20240502 02:09:46.064440 6416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 02:09:46.066572 6430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:46.067535 6416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:46.067641 6416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "dc53186baf544ae4a4a92df94b0a3607"
format_stamp: "Formatted at 2024-05-02 02:09:46 on dist-test-slave-0pw1"
I20240502 02:09:46.067741 6416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:46.126602 6416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:46.127270 6416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:46.127444 6416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:46.128038 6416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:09:46.128947 6416 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:09:46.128998 6416 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:46.129060 6416 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:09:46.129086 6416 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:46.137420 6416 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.196:42299
I20240502 02:09:46.137446 6544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.196:42299 every 8 connection(s)
I20240502 02:09:46.138377 6416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 02:09:46.143803 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 6416
I20240502 02:09:46.143944 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 02:09:46.147599 6545 heartbeater.cc:347] Connected to a master server at 127.27.111.254:36011
I20240502 02:09:46.147815 6545 heartbeater.cc:464] Registering TS with master...
I20240502 02:09:46.148238 6545 heartbeater.cc:511] Master 127.27.111.254:36011 requested a full tablet report, sending...
I20240502 02:09:46.149148 5958 ts_manager.cc:194] Registered new tserver with Master: dc53186baf544ae4a4a92df94b0a3607 (127.27.111.196:42299)
I20240502 02:09:46.149696 5958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.196:44053
I20240502 02:09:46.157852 28095 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 02:09:46.166491 5958 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51884:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 02:09:46.182471 6082 tablet_service.cc:1467] Processing CreateTablet for tablet 8f0db4cff5274c5f9dbe9b807e621f67 (DEFAULT_TABLE table=TestTable [id=19fed4bd98c847d09d4686f6d340e4b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:46.183411 6082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f0db4cff5274c5f9dbe9b807e621f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:46.188733 6558 tablet_bootstrap.cc:492] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207: Bootstrap starting.
I20240502 02:09:46.190750 6558 tablet_bootstrap.cc:654] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:46.191432 6558 log.cc:824] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207: Log is configured to *not* fsync() on all Append() calls
I20240502 02:09:46.193105 6558 tablet_bootstrap.cc:492] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207: No bootstrap required, opened a new log
I20240502 02:09:46.193284 6558 ts_tablet_manager.cc:1367] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 02:09:46.196251 6558 raft_consensus.cc:348] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:46.196440 6558 raft_consensus.cc:374] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:46.196518 6558 raft_consensus.cc:729] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f91f1eaf8349178678a275be055207, State: Initialized, Role: FOLLOWER
I20240502 02:09:46.196950 6558 consensus_queue.cc:260] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:46.197081 6558 raft_consensus.cc:388] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:46.197149 6558 raft_consensus.cc:482] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:46.197259 6558 raft_consensus.cc:3037] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:46.198086 6558 raft_consensus.cc:504] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:46.198446 6558 leader_election.cc:302] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f91f1eaf8349178678a275be055207; no voters:
I20240502 02:09:46.198794 6558 leader_election.cc:288] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:46.198925 6560 raft_consensus.cc:2781] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:46.199159 6560 raft_consensus.cc:686] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207 [term 1 LEADER]: Becoming Leader. State: Replica: 05f91f1eaf8349178678a275be055207, State: Running, Role: LEADER
I20240502 02:09:46.199453 6560 consensus_queue.cc:237] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:46.199527 6148 heartbeater.cc:503] Master 127.27.111.254:36011 was elected leader, sending a full tablet report...
I20240502 02:09:46.199674 6558 ts_tablet_manager.cc:1398] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 02:09:46.202626 5958 catalog_manager.cc:5484] T 8f0db4cff5274c5f9dbe9b807e621f67 P 05f91f1eaf8349178678a275be055207 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f91f1eaf8349178678a275be055207 (127.27.111.193). New cstate: current_term: 1 leader_uuid: "05f91f1eaf8349178678a275be055207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } } }
I20240502 02:09:46.209095 28095 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 02:09:46.210362 28095 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 05f91f1eaf8349178678a275be055207 to finish bootstrapping
I20240502 02:09:46.215296 28095 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f659a9953c2341c48931e531cc59c781 to finish bootstrapping
I20240502 02:09:46.220914 28095 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4fe3bba8e9f14ba79a92f12af08feaca to finish bootstrapping
I20240502 02:09:46.226459 28095 ts_itest-base.cc:233] Waiting for 0 tablets on tserver dc53186baf544ae4a4a92df94b0a3607 to finish bootstrapping
I20240502 02:09:46.231935 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 6283
I20240502 02:09:46.237654 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 6416
I20240502 02:09:46.767988 6280 heartbeater.cc:503] Master 127.27.111.254:36011 was elected leader, sending a full tablet report...
I20240502 02:09:49.994861 5960 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51884:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 02:09:50.001765 6082 tablet_service.cc:1467] Processing CreateTablet for tablet c29272ea50824c2ebe20a132a7e146d4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=79c22255ad944f1a99f6c23696c387a4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:50.002121 6082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c29272ea50824c2ebe20a132a7e146d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:50.004256 6575 tablet_bootstrap.cc:492] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207: Bootstrap starting.
I20240502 02:09:50.005210 6575 tablet_bootstrap.cc:654] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:50.006326 6575 tablet_bootstrap.cc:492] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207: No bootstrap required, opened a new log
I20240502 02:09:50.006476 6575 ts_tablet_manager.cc:1367] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:50.006469 6213 tablet_service.cc:1467] Processing CreateTablet for tablet 3c49494aca3243c6bfc090e818ca1e49 (DEFAULT_TABLE table=rebalance_test_table_0 [id=79c22255ad944f1a99f6c23696c387a4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:50.006482 6212 tablet_service.cc:1467] Processing CreateTablet for tablet c0c0548f291545db9e65dd6c309e9d52 (DEFAULT_TABLE table=rebalance_test_table_0 [id=79c22255ad944f1a99f6c23696c387a4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:50.007033 6575 raft_consensus.cc:348] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.007148 6575 raft_consensus.cc:374] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:50.007185 6575 raft_consensus.cc:729] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f91f1eaf8349178678a275be055207, State: Initialized, Role: FOLLOWER
I20240502 02:09:50.007323 6575 consensus_queue.cc:260] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.007411 6575 raft_consensus.cc:388] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:50.007448 6575 raft_consensus.cc:482] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:50.007500 6575 raft_consensus.cc:3037] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:50.007568 6212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c0548f291545db9e65dd6c309e9d52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:50.007911 6213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c49494aca3243c6bfc090e818ca1e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:50.008306 6575 raft_consensus.cc:504] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.008432 6575 leader_election.cc:302] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f91f1eaf8349178678a275be055207; no voters:
I20240502 02:09:50.008620 6575 leader_election.cc:288] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:50.008715 6577 raft_consensus.cc:2781] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:50.008917 6577 raft_consensus.cc:686] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207 [term 1 LEADER]: Becoming Leader. State: Replica: 05f91f1eaf8349178678a275be055207, State: Running, Role: LEADER
I20240502 02:09:50.008991 6575 ts_tablet_manager.cc:1398] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:50.009107 6577 consensus_queue.cc:237] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.010470 5958 catalog_manager.cc:5484] T c29272ea50824c2ebe20a132a7e146d4 P 05f91f1eaf8349178678a275be055207 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f91f1eaf8349178678a275be055207 (127.27.111.193). New cstate: current_term: 1 leader_uuid: "05f91f1eaf8349178678a275be055207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } } }
I20240502 02:09:50.013945 6582 tablet_bootstrap.cc:492] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781: Bootstrap starting.
I20240502 02:09:50.015769 6582 tablet_bootstrap.cc:654] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:50.016423 6582 log.cc:824] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781: Log is configured to *not* fsync() on all Append() calls
I20240502 02:09:50.017812 6582 tablet_bootstrap.cc:492] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781: No bootstrap required, opened a new log
I20240502 02:09:50.017952 6582 ts_tablet_manager.cc:1367] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 02:09:50.020843 6582 raft_consensus.cc:348] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.021016 6582 raft_consensus.cc:374] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:50.021057 6582 raft_consensus.cc:729] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f659a9953c2341c48931e531cc59c781, State: Initialized, Role: FOLLOWER
I20240502 02:09:50.021425 6582 consensus_queue.cc:260] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.021536 6582 raft_consensus.cc:388] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:50.021581 6582 raft_consensus.cc:482] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:50.021661 6582 raft_consensus.cc:3037] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:50.022421 6582 raft_consensus.cc:504] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.022737 6582 leader_election.cc:302] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f659a9953c2341c48931e531cc59c781; no voters:
I20240502 02:09:50.022991 6582 leader_election.cc:288] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:50.023102 6584 raft_consensus.cc:2781] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:50.023334 6584 raft_consensus.cc:686] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781 [term 1 LEADER]: Becoming Leader. State: Replica: f659a9953c2341c48931e531cc59c781, State: Running, Role: LEADER
I20240502 02:09:50.023669 6584 consensus_queue.cc:237] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.023794 6582 ts_tablet_manager.cc:1398] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 02:09:50.024020 6582 tablet_bootstrap.cc:492] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781: Bootstrap starting.
I20240502 02:09:50.024933 6582 tablet_bootstrap.cc:654] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:50.026592 6582 tablet_bootstrap.cc:492] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781: No bootstrap required, opened a new log
I20240502 02:09:50.026669 6582 ts_tablet_manager.cc:1367] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 02:09:50.026676 5960 catalog_manager.cc:5484] T 3c49494aca3243c6bfc090e818ca1e49 P f659a9953c2341c48931e531cc59c781 reported cstate change: term changed from 0 to 1, leader changed from <none> to f659a9953c2341c48931e531cc59c781 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "f659a9953c2341c48931e531cc59c781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } } }
I20240502 02:09:50.027159 6582 raft_consensus.cc:348] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.027269 6582 raft_consensus.cc:374] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:50.027300 6582 raft_consensus.cc:729] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f659a9953c2341c48931e531cc59c781, State: Initialized, Role: FOLLOWER
I20240502 02:09:50.027426 6582 consensus_queue.cc:260] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.027509 6582 raft_consensus.cc:388] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:50.027550 6582 raft_consensus.cc:482] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:50.027596 6582 raft_consensus.cc:3037] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:50.028355 6582 raft_consensus.cc:504] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.028515 6582 leader_election.cc:302] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f659a9953c2341c48931e531cc59c781; no voters:
I20240502 02:09:50.028617 6582 leader_election.cc:288] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:50.028678 6584 raft_consensus.cc:2781] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:50.028854 6582 ts_tablet_manager.cc:1398] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:50.029265 6584 raft_consensus.cc:686] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781 [term 1 LEADER]: Becoming Leader. State: Replica: f659a9953c2341c48931e531cc59c781, State: Running, Role: LEADER
I20240502 02:09:50.029392 6584 consensus_queue.cc:237] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.030709 5960 catalog_manager.cc:5484] T c0c0548f291545db9e65dd6c309e9d52 P f659a9953c2341c48931e531cc59c781 reported cstate change: term changed from 0 to 1, leader changed from <none> to f659a9953c2341c48931e531cc59c781 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "f659a9953c2341c48931e531cc59c781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } } }
I20240502 02:09:50.201565 5960 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51884:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 02:09:50.206832 6082 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9ff0906ea840afa6580098ebc29d1a (DEFAULT_TABLE table=rebalance_test_table_1 [id=7af1bb88fdfa4f9c97e77ec11be1f032]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:50.206892 6213 tablet_service.cc:1467] Processing CreateTablet for tablet 44a098b00dcb4602a2af66c26e01efa8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7af1bb88fdfa4f9c97e77ec11be1f032]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:50.207124 6081 tablet_service.cc:1467] Processing CreateTablet for tablet 56700062d43b42a8b58e020f4b5078b2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7af1bb88fdfa4f9c97e77ec11be1f032]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:50.207197 6213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a098b00dcb4602a2af66c26e01efa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:50.207212 6082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9ff0906ea840afa6580098ebc29d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:50.207360 6081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56700062d43b42a8b58e020f4b5078b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:50.208977 6582 tablet_bootstrap.cc:492] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781: Bootstrap starting.
I20240502 02:09:50.209455 6575 tablet_bootstrap.cc:492] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207: Bootstrap starting.
I20240502 02:09:50.209903 6582 tablet_bootstrap.cc:654] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:50.210430 6575 tablet_bootstrap.cc:654] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:50.211464 6582 tablet_bootstrap.cc:492] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781: No bootstrap required, opened a new log
I20240502 02:09:50.211530 6575 tablet_bootstrap.cc:492] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207: No bootstrap required, opened a new log
I20240502 02:09:50.211555 6582 ts_tablet_manager.cc:1367] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 02:09:50.211624 6575 ts_tablet_manager.cc:1367] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:50.211963 6582 raft_consensus.cc:348] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.212069 6582 raft_consensus.cc:374] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:50.212034 6575 raft_consensus.cc:348] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.212108 6575 raft_consensus.cc:374] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:50.212102 6582 raft_consensus.cc:729] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f659a9953c2341c48931e531cc59c781, State: Initialized, Role: FOLLOWER
I20240502 02:09:50.212167 6575 raft_consensus.cc:729] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f91f1eaf8349178678a275be055207, State: Initialized, Role: FOLLOWER
I20240502 02:09:50.212276 6582 consensus_queue.cc:260] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.212301 6575 consensus_queue.cc:260] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.212366 6575 raft_consensus.cc:388] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:50.212370 6582 raft_consensus.cc:388] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:50.212392 6575 raft_consensus.cc:482] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:50.212419 6582 raft_consensus.cc:482] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:50.212431 6575 raft_consensus.cc:3037] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:50.212471 6582 raft_consensus.cc:3037] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:50.213093 6575 raft_consensus.cc:504] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.213124 6582 raft_consensus.cc:504] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.213203 6575 leader_election.cc:302] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f91f1eaf8349178678a275be055207; no voters:
I20240502 02:09:50.213243 6582 leader_election.cc:302] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f659a9953c2341c48931e531cc59c781; no voters:
I20240502 02:09:50.213310 6575 leader_election.cc:288] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:50.213352 6582 leader_election.cc:288] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:50.213397 6578 raft_consensus.cc:2781] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:50.213429 6584 raft_consensus.cc:2781] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:50.213488 6578 raft_consensus.cc:686] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207 [term 1 LEADER]: Becoming Leader. State: Replica: 05f91f1eaf8349178678a275be055207, State: Running, Role: LEADER
I20240502 02:09:50.213539 6584 raft_consensus.cc:686] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781 [term 1 LEADER]: Becoming Leader. State: Replica: f659a9953c2341c48931e531cc59c781, State: Running, Role: LEADER
I20240502 02:09:50.213603 6582 ts_tablet_manager.cc:1398] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 02:09:50.213599 6578 consensus_queue.cc:237] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.213639 6584 consensus_queue.cc:237] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.213860 6575 ts_tablet_manager.cc:1398] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:50.214345 6575 tablet_bootstrap.cc:492] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207: Bootstrap starting.
I20240502 02:09:50.215015 5960 catalog_manager.cc:5484] T 44a098b00dcb4602a2af66c26e01efa8 P f659a9953c2341c48931e531cc59c781 reported cstate change: term changed from 0 to 1, leader changed from <none> to f659a9953c2341c48931e531cc59c781 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "f659a9953c2341c48931e531cc59c781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } } }
I20240502 02:09:50.215348 5956 catalog_manager.cc:5484] T 56700062d43b42a8b58e020f4b5078b2 P 05f91f1eaf8349178678a275be055207 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f91f1eaf8349178678a275be055207 (127.27.111.193). New cstate: current_term: 1 leader_uuid: "05f91f1eaf8349178678a275be055207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } } }
I20240502 02:09:50.216598 6575 tablet_bootstrap.cc:654] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:50.217871 6575 tablet_bootstrap.cc:492] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207: No bootstrap required, opened a new log
I20240502 02:09:50.217953 6575 ts_tablet_manager.cc:1367] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 02:09:50.218534 6575 raft_consensus.cc:348] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.218629 6575 raft_consensus.cc:374] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:50.218669 6575 raft_consensus.cc:729] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f91f1eaf8349178678a275be055207, State: Initialized, Role: FOLLOWER
I20240502 02:09:50.218776 6575 consensus_queue.cc:260] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.218854 6575 raft_consensus.cc:388] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:50.218894 6575 raft_consensus.cc:482] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:50.218935 6575 raft_consensus.cc:3037] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:50.219627 6575 raft_consensus.cc:504] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.219732 6575 leader_election.cc:302] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f91f1eaf8349178678a275be055207; no voters:
I20240502 02:09:50.219818 6575 leader_election.cc:288] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:50.219866 6578 raft_consensus.cc:2781] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:50.219945 6578 raft_consensus.cc:686] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207 [term 1 LEADER]: Becoming Leader. State: Replica: 05f91f1eaf8349178678a275be055207, State: Running, Role: LEADER
I20240502 02:09:50.220036 6575 ts_tablet_manager.cc:1398] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:50.220054 6578 consensus_queue.cc:237] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.221450 5956 catalog_manager.cc:5484] T 0c9ff0906ea840afa6580098ebc29d1a P 05f91f1eaf8349178678a275be055207 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f91f1eaf8349178678a275be055207 (127.27.111.193). New cstate: current_term: 1 leader_uuid: "05f91f1eaf8349178678a275be055207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } } }
I20240502 02:09:50.393313 5956 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51884:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 02:09:50.398653 6213 tablet_service.cc:1467] Processing CreateTablet for tablet a71bd5f90c554900b4d69875d0638918 (DEFAULT_TABLE table=rebalance_test_table_2 [id=834c59519e0b47baaa8c1ee1d5104190]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:50.398671 6212 tablet_service.cc:1467] Processing CreateTablet for tablet 3b02ff83f42549a8b36807cf022d01dd (DEFAULT_TABLE table=rebalance_test_table_2 [id=834c59519e0b47baaa8c1ee1d5104190]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:50.398862 6081 tablet_service.cc:1467] Processing CreateTablet for tablet 5a18ad74746741c19e665f128774c40b (DEFAULT_TABLE table=rebalance_test_table_2 [id=834c59519e0b47baaa8c1ee1d5104190]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:50.399008 6213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a71bd5f90c554900b4d69875d0638918. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:50.399144 6081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a18ad74746741c19e665f128774c40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:50.399263 6212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b02ff83f42549a8b36807cf022d01dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:50.400769 6582 tablet_bootstrap.cc:492] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781: Bootstrap starting.
I20240502 02:09:50.401130 6575 tablet_bootstrap.cc:492] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207: Bootstrap starting.
I20240502 02:09:50.401684 6582 tablet_bootstrap.cc:654] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:50.401902 6575 tablet_bootstrap.cc:654] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:50.402797 6582 tablet_bootstrap.cc:492] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781: No bootstrap required, opened a new log
I20240502 02:09:50.402886 6582 ts_tablet_manager.cc:1367] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 02:09:50.402983 6575 tablet_bootstrap.cc:492] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207: No bootstrap required, opened a new log
I20240502 02:09:50.403072 6575 ts_tablet_manager.cc:1367] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:50.403338 6582 raft_consensus.cc:348] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.403409 6582 raft_consensus.cc:374] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:50.403429 6582 raft_consensus.cc:729] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f659a9953c2341c48931e531cc59c781, State: Initialized, Role: FOLLOWER
I20240502 02:09:50.403509 6575 raft_consensus.cc:348] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.403529 6582 consensus_queue.cc:260] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.403604 6582 raft_consensus.cc:388] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:50.403599 6575 raft_consensus.cc:374] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:50.403636 6582 raft_consensus.cc:482] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:50.403637 6575 raft_consensus.cc:729] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f91f1eaf8349178678a275be055207, State: Initialized, Role: FOLLOWER
I20240502 02:09:50.403674 6582 raft_consensus.cc:3037] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:50.403754 6575 consensus_queue.cc:260] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.403837 6575 raft_consensus.cc:388] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:50.403859 6575 raft_consensus.cc:482] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:50.403888 6575 raft_consensus.cc:3037] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:50.404249 6582 raft_consensus.cc:504] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.404342 6582 leader_election.cc:302] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f659a9953c2341c48931e531cc59c781; no voters:
I20240502 02:09:50.404405 6582 leader_election.cc:288] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:50.404443 6575 raft_consensus.cc:504] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.404516 6586 raft_consensus.cc:2781] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:50.404562 6575 leader_election.cc:302] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f91f1eaf8349178678a275be055207; no voters:
I20240502 02:09:50.404608 6586 raft_consensus.cc:686] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781 [term 1 LEADER]: Becoming Leader. State: Replica: f659a9953c2341c48931e531cc59c781, State: Running, Role: LEADER
I20240502 02:09:50.404664 6582 ts_tablet_manager.cc:1398] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:50.404821 6582 tablet_bootstrap.cc:492] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781: Bootstrap starting.
I20240502 02:09:50.404790 6586 consensus_queue.cc:237] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.405181 6575 leader_election.cc:288] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:50.405246 6578 raft_consensus.cc:2781] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:50.405345 6578 raft_consensus.cc:686] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207 [term 1 LEADER]: Becoming Leader. State: Replica: 05f91f1eaf8349178678a275be055207, State: Running, Role: LEADER
I20240502 02:09:50.405462 6575 ts_tablet_manager.cc:1398] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:50.405448 6578 consensus_queue.cc:237] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.405676 6582 tablet_bootstrap.cc:654] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:50.406917 5960 catalog_manager.cc:5484] T 3b02ff83f42549a8b36807cf022d01dd P f659a9953c2341c48931e531cc59c781 reported cstate change: term changed from 0 to 1, leader changed from <none> to f659a9953c2341c48931e531cc59c781 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "f659a9953c2341c48931e531cc59c781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } } }
I20240502 02:09:50.407104 5956 catalog_manager.cc:5484] T 5a18ad74746741c19e665f128774c40b P 05f91f1eaf8349178678a275be055207 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f91f1eaf8349178678a275be055207 (127.27.111.193). New cstate: current_term: 1 leader_uuid: "05f91f1eaf8349178678a275be055207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } } }
I20240502 02:09:50.407841 6582 tablet_bootstrap.cc:492] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781: No bootstrap required, opened a new log
I20240502 02:09:50.407922 6582 ts_tablet_manager.cc:1367] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 02:09:50.408330 6582 raft_consensus.cc:348] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.408434 6582 raft_consensus.cc:374] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:50.408470 6582 raft_consensus.cc:729] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f659a9953c2341c48931e531cc59c781, State: Initialized, Role: FOLLOWER
I20240502 02:09:50.408630 6582 consensus_queue.cc:260] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.408722 6582 raft_consensus.cc:388] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:50.408764 6582 raft_consensus.cc:482] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:50.408811 6582 raft_consensus.cc:3037] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:50.409859 6582 raft_consensus.cc:504] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.409988 6582 leader_election.cc:302] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f659a9953c2341c48931e531cc59c781; no voters:
I20240502 02:09:50.410080 6582 leader_election.cc:288] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:50.410135 6586 raft_consensus.cc:2781] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:50.410238 6586 raft_consensus.cc:686] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781 [term 1 LEADER]: Becoming Leader. State: Replica: f659a9953c2341c48931e531cc59c781, State: Running, Role: LEADER
I20240502 02:09:50.410349 6586 consensus_queue.cc:237] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.410907 6582 ts_tablet_manager.cc:1398] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 02:09:50.412004 5960 catalog_manager.cc:5484] T a71bd5f90c554900b4d69875d0638918 P f659a9953c2341c48931e531cc59c781 reported cstate change: term changed from 0 to 1, leader changed from <none> to f659a9953c2341c48931e531cc59c781 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "f659a9953c2341c48931e531cc59c781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } } }
I20240502 02:09:50.573700 5960 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51884:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 02:09:50.579094 6081 tablet_service.cc:1467] Processing CreateTablet for tablet fcae2640922b44cd99bf8e494a541329 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6ffa0143a7a7496c98a38a57537c5cb2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:50.579190 6082 tablet_service.cc:1467] Processing CreateTablet for tablet 03a9154ae9d846ddb49ec18d8865c8fc (DEFAULT_TABLE table=rebalance_test_table_3 [id=6ffa0143a7a7496c98a38a57537c5cb2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:50.579394 6081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcae2640922b44cd99bf8e494a541329. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:50.579572 6082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03a9154ae9d846ddb49ec18d8865c8fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:50.579838 6213 tablet_service.cc:1467] Processing CreateTablet for tablet a36655d311e74e798c8f52cfe0381db7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=6ffa0143a7a7496c98a38a57537c5cb2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:50.580164 6213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36655d311e74e798c8f52cfe0381db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:50.581576 6575 tablet_bootstrap.cc:492] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207: Bootstrap starting.
I20240502 02:09:50.582252 6582 tablet_bootstrap.cc:492] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781: Bootstrap starting.
I20240502 02:09:50.582469 6575 tablet_bootstrap.cc:654] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:50.583055 6582 tablet_bootstrap.cc:654] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:50.583782 6575 tablet_bootstrap.cc:492] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207: No bootstrap required, opened a new log
I20240502 02:09:50.583853 6575 ts_tablet_manager.cc:1367] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 02:09:50.584239 6582 tablet_bootstrap.cc:492] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781: No bootstrap required, opened a new log
I20240502 02:09:50.584203 6575 raft_consensus.cc:348] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.584281 6575 raft_consensus.cc:374] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:50.584307 6575 raft_consensus.cc:729] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f91f1eaf8349178678a275be055207, State: Initialized, Role: FOLLOWER
I20240502 02:09:50.584319 6582 ts_tablet_manager.cc:1367] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:50.584414 6575 consensus_queue.cc:260] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.584484 6575 raft_consensus.cc:388] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:50.584517 6575 raft_consensus.cc:482] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:50.584556 6575 raft_consensus.cc:3037] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:50.584765 6582 raft_consensus.cc:348] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.584851 6582 raft_consensus.cc:374] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:50.584890 6582 raft_consensus.cc:729] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f659a9953c2341c48931e531cc59c781, State: Initialized, Role: FOLLOWER
I20240502 02:09:50.585014 6582 consensus_queue.cc:260] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.585093 6582 raft_consensus.cc:388] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:50.585132 6582 raft_consensus.cc:482] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:50.585186 6582 raft_consensus.cc:3037] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:50.585358 6575 raft_consensus.cc:504] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.585489 6575 leader_election.cc:302] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f91f1eaf8349178678a275be055207; no voters:
I20240502 02:09:50.585762 6577 raft_consensus.cc:2781] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:50.585836 6577 raft_consensus.cc:686] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207 [term 1 LEADER]: Becoming Leader. State: Replica: 05f91f1eaf8349178678a275be055207, State: Running, Role: LEADER
I20240502 02:09:50.585937 6577 consensus_queue.cc:237] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.585992 6582 raft_consensus.cc:504] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.586112 6582 leader_election.cc:302] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f659a9953c2341c48931e531cc59c781; no voters:
I20240502 02:09:50.586207 6582 leader_election.cc:288] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:50.586247 6584 raft_consensus.cc:2781] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:50.586367 6575 leader_election.cc:288] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:50.586634 6575 ts_tablet_manager.cc:1398] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 02:09:50.586807 6575 tablet_bootstrap.cc:492] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207: Bootstrap starting.
I20240502 02:09:50.587271 6584 raft_consensus.cc:686] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781 [term 1 LEADER]: Becoming Leader. State: Replica: f659a9953c2341c48931e531cc59c781, State: Running, Role: LEADER
I20240502 02:09:50.587397 6584 consensus_queue.cc:237] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } }
I20240502 02:09:50.587543 6575 tablet_bootstrap.cc:654] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:50.588171 5960 catalog_manager.cc:5484] T 03a9154ae9d846ddb49ec18d8865c8fc P 05f91f1eaf8349178678a275be055207 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f91f1eaf8349178678a275be055207 (127.27.111.193). New cstate: current_term: 1 leader_uuid: "05f91f1eaf8349178678a275be055207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } } }
I20240502 02:09:50.588505 6582 ts_tablet_manager.cc:1398] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 02:09:50.588830 5956 catalog_manager.cc:5484] T a36655d311e74e798c8f52cfe0381db7 P f659a9953c2341c48931e531cc59c781 reported cstate change: term changed from 0 to 1, leader changed from <none> to f659a9953c2341c48931e531cc59c781 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "f659a9953c2341c48931e531cc59c781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f659a9953c2341c48931e531cc59c781" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 33833 } } }
I20240502 02:09:50.589121 6575 tablet_bootstrap.cc:492] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207: No bootstrap required, opened a new log
I20240502 02:09:50.589191 6575 ts_tablet_manager.cc:1367] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 02:09:50.589608 6575 raft_consensus.cc:348] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.589689 6575 raft_consensus.cc:374] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:50.589718 6575 raft_consensus.cc:729] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f91f1eaf8349178678a275be055207, State: Initialized, Role: FOLLOWER
I20240502 02:09:50.589843 6575 consensus_queue.cc:260] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.589928 6575 raft_consensus.cc:388] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:50.589974 6575 raft_consensus.cc:482] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:50.590029 6575 raft_consensus.cc:3037] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:50.590967 6575 raft_consensus.cc:504] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.591089 6575 leader_election.cc:302] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f91f1eaf8349178678a275be055207; no voters:
I20240502 02:09:50.591178 6575 leader_election.cc:288] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:50.591298 6577 raft_consensus.cc:2781] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:50.591365 6577 raft_consensus.cc:686] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207 [term 1 LEADER]: Becoming Leader. State: Replica: 05f91f1eaf8349178678a275be055207, State: Running, Role: LEADER
I20240502 02:09:50.591390 6575 ts_tablet_manager.cc:1398] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:50.591464 6577 consensus_queue.cc:237] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } }
I20240502 02:09:50.593658 5960 catalog_manager.cc:5484] T fcae2640922b44cd99bf8e494a541329 P 05f91f1eaf8349178678a275be055207 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f91f1eaf8349178678a275be055207 (127.27.111.193). New cstate: current_term: 1 leader_uuid: "05f91f1eaf8349178678a275be055207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f91f1eaf8349178678a275be055207" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 44903 } } }
I20240502 02:09:50.755664 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.195:37645
--local_ip_for_outbound_sockets=127.27.111.195
--tserver_master_addrs=127.27.111.254:36011
--webserver_port=36125
--webserver_interface=127.27.111.195
--builtin_ntp_servers=127.27.111.212:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 02:09:50.865692 6684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:50.865963 6684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:50.866047 6684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:50.869114 6684 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 02:09:50.869201 6684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:50.869307 6684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.195
I20240502 02:09:50.872996 6684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.195:37645
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.111.195
--webserver_port=36125
--tserver_master_addrs=127.27.111.254:36011
--never_fsync=true
--heap_profile_path=/tmp/kudu.6684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.195
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:50.873972 6684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:50.875133 6684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:50.883333 6689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:50.884045 6692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:50.884585 6690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:50.885321 6684 server_base.cc:1034] running on GCE node
I20240502 02:09:50.885746 6684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:50.886482 6684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:50.887652 6684 hybrid_clock.cc:648] HybridClock initialized: now 1714615790887634 us; error 36 us; skew 500 ppm
I20240502 02:09:50.889658 6684 webserver.cc:460] Webserver started at http://127.27.111.195:36125/ using document root <none> and password file <none>
I20240502 02:09:50.890210 6684 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:50.890278 6684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:50.893687 6684 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:50.895607 6698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:50.896592 6684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 02:09:50.896684 6684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4fe3bba8e9f14ba79a92f12af08feaca"
format_stamp: "Formatted at 2024-05-02 02:09:45 on dist-test-slave-0pw1"
I20240502 02:09:50.897099 6684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:50.912674 6684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:50.913285 6684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:50.913477 6684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:50.914026 6684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:09:50.914971 6684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:09:50.915022 6684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:50.915073 6684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:09:50.915122 6684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:50.923537 6684 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.195:37645
I20240502 02:09:50.923568 6811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.195:37645 every 8 connection(s)
I20240502 02:09:50.924590 6684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 02:09:50.931983 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 6684
I20240502 02:09:50.932549 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.196:42299
--local_ip_for_outbound_sockets=127.27.111.196
--tserver_master_addrs=127.27.111.254:36011
--webserver_port=45045
--webserver_interface=127.27.111.196
--builtin_ntp_servers=127.27.111.212:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 02:09:50.933298 6812 heartbeater.cc:347] Connected to a master server at 127.27.111.254:36011
I20240502 02:09:50.933574 6812 heartbeater.cc:464] Registering TS with master...
I20240502 02:09:50.934149 6812 heartbeater.cc:511] Master 127.27.111.254:36011 requested a full tablet report, sending...
I20240502 02:09:50.935376 5960 ts_manager.cc:194] Re-registered known tserver with Master: 4fe3bba8e9f14ba79a92f12af08feaca (127.27.111.195:37645)
I20240502 02:09:50.936236 5960 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.195:47137
W20240502 02:09:51.045300 6815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:51.045584 6815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:51.045637 6815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:51.048727 6815 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 02:09:51.048799 6815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:51.048866 6815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.196
I20240502 02:09:51.052420 6815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.196:42299
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.111.196
--webserver_port=45045
--tserver_master_addrs=127.27.111.254:36011
--never_fsync=true
--heap_profile_path=/tmp/kudu.6815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.196
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:51.053316 6815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:51.054535 6815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:51.063387 6821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:51.063946 6823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:51.064299 6820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:51.065106 6815 server_base.cc:1034] running on GCE node
I20240502 02:09:51.065578 6815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:51.066319 6815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:51.067498 6815 hybrid_clock.cc:648] HybridClock initialized: now 1714615791067478 us; error 37 us; skew 500 ppm
I20240502 02:09:51.069550 6815 webserver.cc:460] Webserver started at http://127.27.111.196:45045/ using document root <none> and password file <none>
I20240502 02:09:51.070307 6815 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:51.070441 6815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:51.074070 6815 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 02:09:51.076103 6829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:51.077111 6815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 02:09:51.077230 6815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "dc53186baf544ae4a4a92df94b0a3607"
format_stamp: "Formatted at 2024-05-02 02:09:46 on dist-test-slave-0pw1"
I20240502 02:09:51.077682 6815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:51.098829 6815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:51.099433 6815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:51.099606 6815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:51.100127 6815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:09:51.101029 6815 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:09:51.101080 6815 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:51.101141 6815 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:09:51.101166 6815 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:51.109623 6815 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.196:42299
I20240502 02:09:51.109632 6943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.196:42299 every 8 connection(s)
I20240502 02:09:51.110706 6815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 02:09:51.119472 6944 heartbeater.cc:347] Connected to a master server at 127.27.111.254:36011
I20240502 02:09:51.119701 6944 heartbeater.cc:464] Registering TS with master...
I20240502 02:09:51.119920 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 6815
I20240502 02:09:51.120165 6944 heartbeater.cc:511] Master 127.27.111.254:36011 requested a full tablet report, sending...
I20240502 02:09:51.121191 5960 ts_manager.cc:194] Re-registered known tserver with Master: dc53186baf544ae4a4a92df94b0a3607 (127.27.111.196:42299)
I20240502 02:09:51.121920 5960 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.196:49211
I20240502 02:09:51.314379 6081 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:09:51.314584 6746 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:51.315348 6877 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:51.315783 6213 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:09:51.421273 6877 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:51.421396 6081 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:09:51.421406 6213 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:09:51.423206 6746 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:51.543841 6081 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:09:51.543910 6877 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:51.544625 6213 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:09:51.545523 6746 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:51.650565 6746 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:51.651168 6213 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:09:51.651618 6877 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:51.651952 6081 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
1c62d235e6c14ce1ba75889238494eaa | 127.27.111.254:36011 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.27.111.212:38575 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
05f91f1eaf8349178678a275be055207 | 127.27.111.193:44903 | HEALTHY | <none> | 7 | 0
4fe3bba8e9f14ba79a92f12af08feaca | 127.27.111.195:37645 | HEALTHY | <none> | 0 | 0
dc53186baf544ae4a4a92df94b0a3607 | 127.27.111.196:42299 | HEALTHY | <none> | 0 | 0
f659a9953c2341c48931e531cc59c781 | 127.27.111.194:33833 | HEALTHY | <none> | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.27.111.193 | experimental | 127.27.111.193:44903
local_ip_for_outbound_sockets | 127.27.111.194 | experimental | 127.27.111.194:33833
local_ip_for_outbound_sockets | 127.27.111.195 | experimental | 127.27.111.195:37645
local_ip_for_outbound_sockets | 127.27.111.196 | experimental | 127.27.111.196:42299
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.27.111.193:44903
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.27.111.194:33833
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.27.111.195:37645
server_dump_info_path | /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.27.111.196:42299
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.27.111.212:38575 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 6
Third Quartile | 7
Maximum | 7
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 02:09:51.678212 28095 log_verifier.cc:126] Checking tablet 03a9154ae9d846ddb49ec18d8865c8fc
I20240502 02:09:51.686147 28095 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 03a9154ae9d846ddb49ec18d8865c8fc
I20240502 02:09:51.686220 28095 log_verifier.cc:126] Checking tablet 0c9ff0906ea840afa6580098ebc29d1a
I20240502 02:09:51.693686 28095 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 0c9ff0906ea840afa6580098ebc29d1a
I20240502 02:09:51.693764 28095 log_verifier.cc:126] Checking tablet 3b02ff83f42549a8b36807cf022d01dd
I20240502 02:09:51.701272 28095 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 3b02ff83f42549a8b36807cf022d01dd
I20240502 02:09:51.701339 28095 log_verifier.cc:126] Checking tablet 3c49494aca3243c6bfc090e818ca1e49
I20240502 02:09:51.708900 28095 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 3c49494aca3243c6bfc090e818ca1e49
I20240502 02:09:51.708972 28095 log_verifier.cc:126] Checking tablet 44a098b00dcb4602a2af66c26e01efa8
I20240502 02:09:51.716429 28095 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 44a098b00dcb4602a2af66c26e01efa8
I20240502 02:09:51.716508 28095 log_verifier.cc:126] Checking tablet 56700062d43b42a8b58e020f4b5078b2
I20240502 02:09:51.723704 28095 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 56700062d43b42a8b58e020f4b5078b2
I20240502 02:09:51.723773 28095 log_verifier.cc:126] Checking tablet 5a18ad74746741c19e665f128774c40b
I20240502 02:09:51.731091 28095 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 5a18ad74746741c19e665f128774c40b
I20240502 02:09:51.731160 28095 log_verifier.cc:126] Checking tablet 8f0db4cff5274c5f9dbe9b807e621f67
I20240502 02:09:51.738701 28095 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 8f0db4cff5274c5f9dbe9b807e621f67
I20240502 02:09:51.738772 28095 log_verifier.cc:126] Checking tablet a36655d311e74e798c8f52cfe0381db7
I20240502 02:09:51.746241 28095 log_verifier.cc:177] Verified matching terms for 1 ops in tablet a36655d311e74e798c8f52cfe0381db7
I20240502 02:09:51.746315 28095 log_verifier.cc:126] Checking tablet a71bd5f90c554900b4d69875d0638918
I20240502 02:09:51.753890 28095 log_verifier.cc:177] Verified matching terms for 2 ops in tablet a71bd5f90c554900b4d69875d0638918
I20240502 02:09:51.753957 28095 log_verifier.cc:126] Checking tablet c0c0548f291545db9e65dd6c309e9d52
I20240502 02:09:51.761525 28095 log_verifier.cc:177] Verified matching terms for 1 ops in tablet c0c0548f291545db9e65dd6c309e9d52
I20240502 02:09:51.761595 28095 log_verifier.cc:126] Checking tablet c29272ea50824c2ebe20a132a7e146d4
I20240502 02:09:51.768972 28095 log_verifier.cc:177] Verified matching terms for 1 ops in tablet c29272ea50824c2ebe20a132a7e146d4
I20240502 02:09:51.769059 28095 log_verifier.cc:126] Checking tablet fcae2640922b44cd99bf8e494a541329
I20240502 02:09:51.776590 28095 log_verifier.cc:177] Verified matching terms for 2 ops in tablet fcae2640922b44cd99bf8e494a541329
I20240502 02:09:51.777374 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 6020
I20240502 02:09:51.784430 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 6151
I20240502 02:09:51.792588 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 6684
I20240502 02:09:51.799149 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 6815
I20240502 02:09:51.804903 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 5927
2024-05-02T02:09:51Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0 (6694 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (6694 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2
I20240502 02:09:51.839865 28095 test_util.cc:268] Using random seed: 422767219
I20240502 02:09:51.840704 28095 ts_itest-base.cc:112] Starting cluster with:
I20240502 02:09:51.840744 28095 ts_itest-base.cc:113] --------------
I20240502 02:09:51.840755 28095 ts_itest-base.cc:114] 5 tablet servers
I20240502 02:09:51.840766 28095 ts_itest-base.cc:115] 3 replicas per TS
I20240502 02:09:51.840776 28095 ts_itest-base.cc:116] --------------
2024-05-02T02:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T02:09:51Z Disabled control of system clock
I20240502 02:09:51.849854 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.111.254:42347
--webserver_interface=127.27.111.254
--webserver_port=0
--builtin_ntp_servers=127.27.111.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.111.254:42347
--location_mapping_cmd=/tmp/dist-test-task43FDSt/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 02:09:51.961917 7093 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 02:09:51.962234 7093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:51.962261 7093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:51.962299 7093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:51.965510 7093 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:09:51.965605 7093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 02:09:51.965631 7093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:51.965648 7093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 02:09:51.965672 7093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 02:09:51.969580 7093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task43FDSt/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.27.111.254:42347
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.111.254:42347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.111.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:51.970620 7093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:51.971794 7093 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:51.979305 7101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:51.979856 7098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:51.980473 7099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:51.981145 7093 server_base.cc:1034] running on GCE node
I20240502 02:09:51.981702 7093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:51.982715 7093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:51.983881 7093 hybrid_clock.cc:648] HybridClock initialized: now 1714615791983873 us; error 26 us; skew 500 ppm
I20240502 02:09:51.985921 7093 webserver.cc:460] Webserver started at http://127.27.111.254:40931/ using document root <none> and password file <none>
I20240502 02:09:51.986543 7093 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:51.986608 7093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:51.986811 7093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:09:51.988539 7093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f1a81c08615d432aac7882c94c08b46c"
format_stamp: "Formatted at 2024-05-02 02:09:51 on dist-test-slave-0pw1"
I20240502 02:09:51.989044 7093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f1a81c08615d432aac7882c94c08b46c"
format_stamp: "Formatted at 2024-05-02 02:09:51 on dist-test-slave-0pw1"
I20240502 02:09:51.992640 7093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 02:09:51.995079 7107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:51.996189 7093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 02:09:51.996323 7093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f1a81c08615d432aac7882c94c08b46c"
format_stamp: "Formatted at 2024-05-02 02:09:51 on dist-test-slave-0pw1"
I20240502 02:09:51.996424 7093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:52.011121 7093 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:52.011862 7093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:52.012032 7093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:52.018927 7093 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.254:42347
I20240502 02:09:52.018942 7159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.254:42347 every 8 connection(s)
I20240502 02:09:52.019982 7093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 02:09:52.022727 7160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:52.026252 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 7093
I20240502 02:09:52.026451 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 02:09:52.028517 7160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c: Bootstrap starting.
I20240502 02:09:52.030834 7160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:52.031570 7160 log.cc:824] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c: Log is configured to *not* fsync() on all Append() calls
I20240502 02:09:52.033568 7160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c: No bootstrap required, opened a new log
I20240502 02:09:52.036602 7160 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a81c08615d432aac7882c94c08b46c" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 42347 } }
I20240502 02:09:52.036808 7160 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:52.036859 7160 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a81c08615d432aac7882c94c08b46c, State: Initialized, Role: FOLLOWER
I20240502 02:09:52.037302 7160 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a81c08615d432aac7882c94c08b46c" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 42347 } }
I20240502 02:09:52.037456 7160 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:09:52.037505 7160 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:09:52.037585 7160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:52.038321 7160 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a81c08615d432aac7882c94c08b46c" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 42347 } }
I20240502 02:09:52.038661 7160 leader_election.cc:302] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1a81c08615d432aac7882c94c08b46c; no voters:
I20240502 02:09:52.038928 7160 leader_election.cc:288] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:09:52.039108 7165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:52.039353 7165 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [term 1 LEADER]: Becoming Leader. State: Replica: f1a81c08615d432aac7882c94c08b46c, State: Running, Role: LEADER
I20240502 02:09:52.039649 7165 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a81c08615d432aac7882c94c08b46c" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 42347 } }
I20240502 02:09:52.040082 7160 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 02:09:52.041466 7166 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1a81c08615d432aac7882c94c08b46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a81c08615d432aac7882c94c08b46c" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 42347 } } }
I20240502 02:09:52.041604 7166 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [sys.catalog]: This master's current role is: LEADER
I20240502 02:09:52.042011 7177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 02:09:52.041970 7167 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1a81c08615d432aac7882c94c08b46c. Latest consensus state: current_term: 1 leader_uuid: "f1a81c08615d432aac7882c94c08b46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a81c08615d432aac7882c94c08b46c" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 42347 } } }
I20240502 02:09:52.042125 7167 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c [sys.catalog]: This master's current role is: LEADER
I20240502 02:09:52.045953 7177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 02:09:52.051803 7177 catalog_manager.cc:1347] Generated new cluster ID: 46d88bd6e0444dacbf01cbf965541af3
I20240502 02:09:52.051901 7177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 02:09:52.075846 7177 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 02:09:52.077246 7177 catalog_manager.cc:1504] Loading token signing keys...
I20240502 02:09:52.090229 7177 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f1a81c08615d432aac7882c94c08b46c: Generated new TSK 0
I20240502 02:09:52.091185 7177 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 02:09:52.100275 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.193:0
--local_ip_for_outbound_sockets=127.27.111.193
--webserver_interface=127.27.111.193
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42347
--builtin_ntp_servers=127.27.111.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 02:09:52.213608 7185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:52.213884 7185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:52.213963 7185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:52.217221 7185 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:09:52.217315 7185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:52.217427 7185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.193
I20240502 02:09:52.221192 7185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.111.193
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42347
--never_fsync=true
--heap_profile_path=/tmp/kudu.7185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.193
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:52.222245 7185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:52.223683 7185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:52.233168 7193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:52.233558 7190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:52.234517 7191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:52.235247 7185 server_base.cc:1034] running on GCE node
I20240502 02:09:52.235721 7185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:52.236497 7185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:52.237685 7185 hybrid_clock.cc:648] HybridClock initialized: now 1714615792237653 us; error 43 us; skew 500 ppm
I20240502 02:09:52.239854 7185 webserver.cc:460] Webserver started at http://127.27.111.193:43529/ using document root <none> and password file <none>
I20240502 02:09:52.240468 7185 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:52.240538 7185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:52.240742 7185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:09:52.242707 7185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "100889921252449596fc397b10fe903d"
format_stamp: "Formatted at 2024-05-02 02:09:52 on dist-test-slave-0pw1"
I20240502 02:09:52.243160 7185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "100889921252449596fc397b10fe903d"
format_stamp: "Formatted at 2024-05-02 02:09:52 on dist-test-slave-0pw1"
I20240502 02:09:52.246567 7185 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240502 02:09:52.248633 7199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:52.249617 7185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:52.249727 7185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "100889921252449596fc397b10fe903d"
format_stamp: "Formatted at 2024-05-02 02:09:52 on dist-test-slave-0pw1"
I20240502 02:09:52.249826 7185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:52.289606 7185 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:52.290318 7185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:52.290545 7185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:52.291143 7185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:09:52.292090 7185 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:09:52.292140 7185 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:52.292188 7185 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:09:52.292222 7185 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:52.301204 7185 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.193:43115
I20240502 02:09:52.301246 7314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.193:43115 every 8 connection(s)
I20240502 02:09:52.302220 7185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 02:09:52.307308 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 7185
I20240502 02:09:52.307442 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 02:09:52.309551 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.194:0
--local_ip_for_outbound_sockets=127.27.111.194
--webserver_interface=127.27.111.194
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42347
--builtin_ntp_servers=127.27.111.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:09:52.311203 7315 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42347
I20240502 02:09:52.311427 7315 heartbeater.cc:464] Registering TS with master...
I20240502 02:09:52.311991 7315 heartbeater.cc:511] Master 127.27.111.254:42347 requested a full tablet report, sending...
I20240502 02:09:52.362368 7124 ts_manager.cc:194] Registered new tserver with Master: 100889921252449596fc397b10fe903d (127.27.111.193:43115)
I20240502 02:09:52.363878 7124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.193:37307
W20240502 02:09:52.439778 7318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:52.440047 7318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:52.440128 7318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:52.443451 7318 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:09:52.443547 7318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:52.443648 7318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.194
I20240502 02:09:52.447598 7318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.111.194
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42347
--never_fsync=true
--heap_profile_path=/tmp/kudu.7318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.194
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:52.448643 7318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:52.449857 7318 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:52.458860 7327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:52.459420 7324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:52.459960 7325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:52.460708 7318 server_base.cc:1034] running on GCE node
I20240502 02:09:52.461136 7318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:52.461850 7318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:52.462997 7318 hybrid_clock.cc:648] HybridClock initialized: now 1714615792462977 us; error 49 us; skew 500 ppm
I20240502 02:09:52.464996 7318 webserver.cc:460] Webserver started at http://127.27.111.194:44897/ using document root <none> and password file <none>
I20240502 02:09:52.465615 7318 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:52.465677 7318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:52.465885 7318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:09:52.467726 7318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ca5956dd8d174df99a28e38e8a7a2524"
format_stamp: "Formatted at 2024-05-02 02:09:52 on dist-test-slave-0pw1"
I20240502 02:09:52.468226 7318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ca5956dd8d174df99a28e38e8a7a2524"
format_stamp: "Formatted at 2024-05-02 02:09:52 on dist-test-slave-0pw1"
I20240502 02:09:52.471874 7318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240502 02:09:52.474202 7333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:52.475392 7318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:52.475543 7318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ca5956dd8d174df99a28e38e8a7a2524"
format_stamp: "Formatted at 2024-05-02 02:09:52 on dist-test-slave-0pw1"
I20240502 02:09:52.475649 7318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:52.498225 7318 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:52.498963 7318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:52.499141 7318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:52.499755 7318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:09:52.500699 7318 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:09:52.500767 7318 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:52.500830 7318 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:09:52.500865 7318 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:52.509923 7318 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.194:40893
I20240502 02:09:52.509943 7446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.194:40893 every 8 connection(s)
I20240502 02:09:52.510998 7318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 02:09:52.517158 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 7318
I20240502 02:09:52.517293 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 02:09:52.519830 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.195:0
--local_ip_for_outbound_sockets=127.27.111.195
--webserver_interface=127.27.111.195
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42347
--builtin_ntp_servers=127.27.111.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:09:52.520362 7447 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42347
I20240502 02:09:52.520623 7447 heartbeater.cc:464] Registering TS with master...
I20240502 02:09:52.521314 7447 heartbeater.cc:511] Master 127.27.111.254:42347 requested a full tablet report, sending...
I20240502 02:09:52.558290 7124 ts_manager.cc:194] Registered new tserver with Master: ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893)
I20240502 02:09:52.559049 7124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.194:49921
W20240502 02:09:52.634934 7450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:52.635249 7450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:52.635339 7450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:52.640314 7450 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:09:52.640419 7450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:52.640538 7450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.195
I20240502 02:09:52.645056 7450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.111.195
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42347
--never_fsync=true
--heap_profile_path=/tmp/kudu.7450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.195
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:52.646072 7450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:52.647225 7450 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:52.656009 7456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:52.656445 7459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:52.656862 7457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:52.657573 7450 server_base.cc:1034] running on GCE node
I20240502 02:09:52.658018 7450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:52.658751 7450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:52.659930 7450 hybrid_clock.cc:648] HybridClock initialized: now 1714615792659913 us; error 36 us; skew 500 ppm
I20240502 02:09:52.661922 7450 webserver.cc:460] Webserver started at http://127.27.111.195:43591/ using document root <none> and password file <none>
I20240502 02:09:52.662539 7450 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:52.662611 7450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:52.662806 7450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:09:52.664541 7450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "defc209b6ddf4b719d89059a990ecb05"
format_stamp: "Formatted at 2024-05-02 02:09:52 on dist-test-slave-0pw1"
I20240502 02:09:52.665033 7450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "defc209b6ddf4b719d89059a990ecb05"
format_stamp: "Formatted at 2024-05-02 02:09:52 on dist-test-slave-0pw1"
I20240502 02:09:52.668594 7450 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 02:09:52.670928 7465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:52.672012 7450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 02:09:52.672142 7450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "defc209b6ddf4b719d89059a990ecb05"
format_stamp: "Formatted at 2024-05-02 02:09:52 on dist-test-slave-0pw1"
I20240502 02:09:52.672238 7450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:52.687471 7450 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:52.688100 7450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:52.688261 7450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:52.688797 7450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:09:52.689687 7450 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:09:52.689733 7450 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:52.689787 7450 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:09:52.689819 7450 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:52.698881 7450 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.195:40407
I20240502 02:09:52.698920 7579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.195:40407 every 8 connection(s)
I20240502 02:09:52.699874 7450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 02:09:52.706720 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 7450
I20240502 02:09:52.706861 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 02:09:52.709280 7580 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42347
I20240502 02:09:52.709393 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.196:0
--local_ip_for_outbound_sockets=127.27.111.196
--webserver_interface=127.27.111.196
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42347
--builtin_ntp_servers=127.27.111.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:09:52.709586 7580 heartbeater.cc:464] Registering TS with master...
I20240502 02:09:52.710145 7580 heartbeater.cc:511] Master 127.27.111.254:42347 requested a full tablet report, sending...
I20240502 02:09:52.752317 7124 ts_manager.cc:194] Registered new tserver with Master: defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:09:52.753139 7124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.195:43225
W20240502 02:09:52.829174 7583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:52.829437 7583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:52.829483 7583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:52.832650 7583 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:09:52.832732 7583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:52.832825 7583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.196
I20240502 02:09:52.836621 7583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.111.196
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42347
--never_fsync=true
--heap_profile_path=/tmp/kudu.7583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.196
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:52.837633 7583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:52.838866 7583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:52.847533 7590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:52.848011 7592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:52.848743 7589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:52.849340 7583 server_base.cc:1034] running on GCE node
I20240502 02:09:52.849726 7583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:52.850438 7583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:52.851613 7583 hybrid_clock.cc:648] HybridClock initialized: now 1714615792851603 us; error 26 us; skew 500 ppm
I20240502 02:09:52.853605 7583 webserver.cc:460] Webserver started at http://127.27.111.196:33443/ using document root <none> and password file <none>
I20240502 02:09:52.854197 7583 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:52.854259 7583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:52.854501 7583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:09:52.856249 7583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "efabf88707da4c13b7ec85bbfd8a89da"
format_stamp: "Formatted at 2024-05-02 02:09:52 on dist-test-slave-0pw1"
I20240502 02:09:52.856720 7583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "efabf88707da4c13b7ec85bbfd8a89da"
format_stamp: "Formatted at 2024-05-02 02:09:52 on dist-test-slave-0pw1"
I20240502 02:09:52.860261 7583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 02:09:52.862504 7598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:52.863521 7583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 02:09:52.863646 7583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "efabf88707da4c13b7ec85bbfd8a89da"
format_stamp: "Formatted at 2024-05-02 02:09:52 on dist-test-slave-0pw1"
I20240502 02:09:52.863740 7583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:52.892345 7583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:52.893054 7583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:52.893229 7583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:52.893842 7583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:09:52.894804 7583 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:09:52.894855 7583 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:52.894935 7583 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:09:52.894965 7583 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:52.904089 7583 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.196:45431
I20240502 02:09:52.904117 7713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.196:45431 every 8 connection(s)
I20240502 02:09:52.905079 7583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 02:09:52.906442 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 7583
I20240502 02:09:52.906551 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 02:09:52.908731 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.197:0
--local_ip_for_outbound_sockets=127.27.111.197
--webserver_interface=127.27.111.197
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42347
--builtin_ntp_servers=127.27.111.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:09:52.914742 7714 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42347
I20240502 02:09:52.915069 7714 heartbeater.cc:464] Registering TS with master...
I20240502 02:09:52.915872 7714 heartbeater.cc:511] Master 127.27.111.254:42347 requested a full tablet report, sending...
I20240502 02:09:52.955705 7124 ts_manager.cc:194] Registered new tserver with Master: efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196:45431)
I20240502 02:09:52.956725 7124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.196:41641
W20240502 02:09:53.031708 7717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:53.031935 7717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:53.031999 7717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:53.035256 7717 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:09:53.035337 7717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:53.035415 7717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.197
I20240502 02:09:53.039211 7717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.111.197
--webserver_port=0
--tserver_master_addrs=127.27.111.254:42347
--never_fsync=true
--heap_profile_path=/tmp/kudu.7717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.197
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:53.040163 7717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:53.041466 7717 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:53.050287 7723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:53.050848 7726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:53.051223 7724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:53.051951 7717 server_base.cc:1034] running on GCE node
I20240502 02:09:53.052395 7717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:53.053114 7717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:53.054275 7717 hybrid_clock.cc:648] HybridClock initialized: now 1714615793054255 us; error 36 us; skew 500 ppm
I20240502 02:09:53.056326 7717 webserver.cc:460] Webserver started at http://127.27.111.197:37183/ using document root <none> and password file <none>
I20240502 02:09:53.056941 7717 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:53.057006 7717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:53.057309 7717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:09:53.059211 7717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "85e53716ea13433289fe6435b6693a5b"
format_stamp: "Formatted at 2024-05-02 02:09:53 on dist-test-slave-0pw1"
I20240502 02:09:53.059738 7717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "85e53716ea13433289fe6435b6693a5b"
format_stamp: "Formatted at 2024-05-02 02:09:53 on dist-test-slave-0pw1"
I20240502 02:09:53.063431 7717 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 02:09:53.065825 7732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:53.066941 7717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 02:09:53.067188 7717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "85e53716ea13433289fe6435b6693a5b"
format_stamp: "Formatted at 2024-05-02 02:09:53 on dist-test-slave-0pw1"
I20240502 02:09:53.067318 7717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:53.102363 7717 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:53.103137 7717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:53.103327 7717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:53.103929 7717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:09:53.104933 7717 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:09:53.104985 7717 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:53.105027 7717 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:09:53.105042 7717 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:53.114017 7717 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.197:36289
I20240502 02:09:53.114034 7845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.197:36289 every 8 connection(s)
I20240502 02:09:53.115160 7717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 02:09:53.124521 7846 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42347
I20240502 02:09:53.124792 7846 heartbeater.cc:464] Registering TS with master...
I20240502 02:09:53.125367 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 7717
I20240502 02:09:53.125499 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 02:09:53.125468 7846 heartbeater.cc:511] Master 127.27.111.254:42347 requested a full tablet report, sending...
I20240502 02:09:53.161471 7123 ts_manager.cc:194] Registered new tserver with Master: 85e53716ea13433289fe6435b6693a5b (127.27.111.197:36289)
I20240502 02:09:53.162214 7123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.197:58771
I20240502 02:09:53.166867 28095 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 02:09:53.177191 28095 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 02:09:53.177285 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 7717
I20240502 02:09:53.183364 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 7583
I20240502 02:09:53.367024 7315 heartbeater.cc:503] Master 127.27.111.254:42347 was elected leader, sending a full tablet report...
I20240502 02:09:53.561843 7447 heartbeater.cc:503] Master 127.27.111.254:42347 was elected leader, sending a full tablet report...
I20240502 02:09:53.755601 7580 heartbeater.cc:503] Master 127.27.111.254:42347 was elected leader, sending a full tablet report...
I20240502 02:09:56.941092 7123 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55280:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 02:09:56.944463 7123 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 02:09:56.961969 7246 tablet_service.cc:1467] Processing CreateTablet for tablet e7b56f82e754429e8cc8fc6d2b78ce8e (DEFAULT_TABLE table=rebalance_test_table_0 [id=424baa04ef41432ebdec6d07ed41cd5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:56.961891 7511 tablet_service.cc:1467] Processing CreateTablet for tablet a7d090773bb34c6eb370f39ffae4e4ea (DEFAULT_TABLE table=rebalance_test_table_0 [id=424baa04ef41432ebdec6d07ed41cd5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:56.962323 7245 tablet_service.cc:1467] Processing CreateTablet for tablet a7d090773bb34c6eb370f39ffae4e4ea (DEFAULT_TABLE table=rebalance_test_table_0 [id=424baa04ef41432ebdec6d07ed41cd5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:56.962759 7511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d090773bb34c6eb370f39ffae4e4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:56.962759 7246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b56f82e754429e8cc8fc6d2b78ce8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:56.963310 7245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d090773bb34c6eb370f39ffae4e4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:56.961694 7247 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec2a56deaf64463aa75024fb5fa32f4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=424baa04ef41432ebdec6d07ed41cd5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:56.964413 7247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec2a56deaf64463aa75024fb5fa32f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:56.964489 7381 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec2a56deaf64463aa75024fb5fa32f4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=424baa04ef41432ebdec6d07ed41cd5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:56.965420 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec2a56deaf64463aa75024fb5fa32f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:56.966722 7380 tablet_service.cc:1467] Processing CreateTablet for tablet e7b56f82e754429e8cc8fc6d2b78ce8e (DEFAULT_TABLE table=rebalance_test_table_0 [id=424baa04ef41432ebdec6d07ed41cd5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:56.967168 7379 tablet_service.cc:1467] Processing CreateTablet for tablet a7d090773bb34c6eb370f39ffae4e4ea (DEFAULT_TABLE table=rebalance_test_table_0 [id=424baa04ef41432ebdec6d07ed41cd5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:56.967581 7512 tablet_service.cc:1467] Processing CreateTablet for tablet e7b56f82e754429e8cc8fc6d2b78ce8e (DEFAULT_TABLE table=rebalance_test_table_0 [id=424baa04ef41432ebdec6d07ed41cd5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:56.967840 7512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b56f82e754429e8cc8fc6d2b78ce8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:56.961891 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec2a56deaf64463aa75024fb5fa32f4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=424baa04ef41432ebdec6d07ed41cd5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:56.968700 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec2a56deaf64463aa75024fb5fa32f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:56.970222 7866 tablet_bootstrap.cc:492] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05: Bootstrap starting.
I20240502 02:09:56.971115 7380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b56f82e754429e8cc8fc6d2b78ce8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:56.971500 7379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d090773bb34c6eb370f39ffae4e4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:56.972342 7866 tablet_bootstrap.cc:654] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:56.973210 7866 log.cc:824] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05: Log is configured to *not* fsync() on all Append() calls
I20240502 02:09:56.974076 7865 tablet_bootstrap.cc:492] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Bootstrap starting.
I20240502 02:09:56.975056 7866 tablet_bootstrap.cc:492] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05: No bootstrap required, opened a new log
I20240502 02:09:56.975241 7866 ts_tablet_manager.cc:1367] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 02:09:56.976178 7865 tablet_bootstrap.cc:654] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:56.977025 7865 log.cc:824] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Log is configured to *not* fsync() on all Append() calls
I20240502 02:09:56.977604 7868 tablet_bootstrap.cc:492] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524: Bootstrap starting.
I20240502 02:09:56.978755 7865 tablet_bootstrap.cc:492] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: No bootstrap required, opened a new log
I20240502 02:09:56.978925 7865 ts_tablet_manager.cc:1367] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 02:09:56.979048 7866 raft_consensus.cc:348] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:56.979259 7866 raft_consensus.cc:374] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:56.979344 7866 raft_consensus.cc:729] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: defc209b6ddf4b719d89059a990ecb05, State: Initialized, Role: FOLLOWER
I20240502 02:09:56.979815 7868 tablet_bootstrap.cc:654] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:56.979796 7866 consensus_queue.cc:260] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:56.980787 7866 ts_tablet_manager.cc:1398] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 02:09:56.981050 7868 log.cc:824] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524: Log is configured to *not* fsync() on all Append() calls
I20240502 02:09:56.981964 7866 tablet_bootstrap.cc:492] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05: Bootstrap starting.
I20240502 02:09:56.982499 7865 raft_consensus.cc:348] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } }
I20240502 02:09:56.982726 7865 raft_consensus.cc:374] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:56.982811 7866 tablet_bootstrap.cc:654] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:56.982846 7865 raft_consensus.cc:729] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100889921252449596fc397b10fe903d, State: Initialized, Role: FOLLOWER
I20240502 02:09:56.983327 7865 consensus_queue.cc:260] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } }
I20240502 02:09:56.984632 7868 tablet_bootstrap.cc:492] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524: No bootstrap required, opened a new log
I20240502 02:09:56.985082 7866 tablet_bootstrap.cc:492] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05: No bootstrap required, opened a new log
I20240502 02:09:56.985169 7866 ts_tablet_manager.cc:1367] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 02:09:56.985610 7866 raft_consensus.cc:348] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } }
I20240502 02:09:56.985718 7866 raft_consensus.cc:374] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:56.985750 7866 raft_consensus.cc:729] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: defc209b6ddf4b719d89059a990ecb05, State: Initialized, Role: FOLLOWER
I20240502 02:09:56.985868 7866 consensus_queue.cc:260] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } }
I20240502 02:09:56.986089 7866 ts_tablet_manager.cc:1398] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:09:56.986122 7865 ts_tablet_manager.cc:1398] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 02:09:56.986207 7866 tablet_bootstrap.cc:492] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05: Bootstrap starting.
I20240502 02:09:56.986424 7865 tablet_bootstrap.cc:492] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d: Bootstrap starting.
I20240502 02:09:56.987118 7866 tablet_bootstrap.cc:654] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:56.987326 7865 tablet_bootstrap.cc:654] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:56.988284 7868 ts_tablet_manager.cc:1367] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 02:09:56.989125 7865 tablet_bootstrap.cc:492] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d: No bootstrap required, opened a new log
I20240502 02:09:56.989202 7865 ts_tablet_manager.cc:1367] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 02:09:56.989667 7865 raft_consensus.cc:348] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:56.989759 7865 raft_consensus.cc:374] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:56.989780 7865 raft_consensus.cc:729] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100889921252449596fc397b10fe903d, State: Initialized, Role: FOLLOWER
I20240502 02:09:56.989905 7865 consensus_queue.cc:260] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:56.990160 7865 ts_tablet_manager.cc:1398] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:09:56.990379 7865 tablet_bootstrap.cc:492] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d: Bootstrap starting.
I20240502 02:09:56.991194 7865 tablet_bootstrap.cc:654] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:56.991501 7868 raft_consensus.cc:348] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:56.991845 7868 raft_consensus.cc:374] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:56.992316 7866 tablet_bootstrap.cc:492] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05: No bootstrap required, opened a new log
I20240502 02:09:56.992349 7865 tablet_bootstrap.cc:492] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d: No bootstrap required, opened a new log
I20240502 02:09:56.992442 7866 ts_tablet_manager.cc:1367] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 02:09:56.992441 7865 ts_tablet_manager.cc:1367] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 02:09:56.992911 7865 raft_consensus.cc:348] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:56.992993 7865 raft_consensus.cc:374] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:56.993014 7865 raft_consensus.cc:729] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100889921252449596fc397b10fe903d, State: Initialized, Role: FOLLOWER
I20240502 02:09:56.993126 7865 consensus_queue.cc:260] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:56.993156 7866 raft_consensus.cc:348] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:56.993234 7866 raft_consensus.cc:374] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:56.993263 7866 raft_consensus.cc:729] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: defc209b6ddf4b719d89059a990ecb05, State: Initialized, Role: FOLLOWER
I20240502 02:09:56.993423 7865 ts_tablet_manager.cc:1398] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:09:56.993439 7866 consensus_queue.cc:260] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:56.993685 7866 ts_tablet_manager.cc:1398] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:09:56.991956 7868 raft_consensus.cc:729] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5956dd8d174df99a28e38e8a7a2524, State: Initialized, Role: FOLLOWER
I20240502 02:09:56.994369 7868 consensus_queue.cc:260] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:56.995483 7868 ts_tablet_manager.cc:1398] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20240502 02:09:56.995898 7868 tablet_bootstrap.cc:492] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524: Bootstrap starting.
I20240502 02:09:56.996804 7868 tablet_bootstrap.cc:654] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:56.998133 7868 tablet_bootstrap.cc:492] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524: No bootstrap required, opened a new log
I20240502 02:09:56.998219 7868 ts_tablet_manager.cc:1367] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 02:09:56.998804 7868 raft_consensus.cc:348] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } }
I20240502 02:09:56.998924 7868 raft_consensus.cc:374] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:56.999008 7868 raft_consensus.cc:729] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5956dd8d174df99a28e38e8a7a2524, State: Initialized, Role: FOLLOWER
I20240502 02:09:56.999146 7868 consensus_queue.cc:260] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } }
I20240502 02:09:56.999411 7868 ts_tablet_manager.cc:1398] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:09:56.999642 7868 tablet_bootstrap.cc:492] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524: Bootstrap starting.
I20240502 02:09:57.000540 7868 tablet_bootstrap.cc:654] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:57.001662 7868 tablet_bootstrap.cc:492] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524: No bootstrap required, opened a new log
I20240502 02:09:57.001757 7868 ts_tablet_manager.cc:1367] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 02:09:57.002138 7868 raft_consensus.cc:348] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.002221 7868 raft_consensus.cc:374] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:57.002241 7868 raft_consensus.cc:729] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5956dd8d174df99a28e38e8a7a2524, State: Initialized, Role: FOLLOWER
I20240502 02:09:57.002342 7868 consensus_queue.cc:260] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.002622 7868 ts_tablet_manager.cc:1398] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240502 02:09:57.015610 7448 tablet.cc:2369] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 02:09:57.040364 7870 raft_consensus.cc:482] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:09:57.040614 7870 raft_consensus.cc:504] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } }
I20240502 02:09:57.041810 7870 leader_election.cc:288] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 100889921252449596fc397b10fe903d (127.27.111.193:43115), ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893)
I20240502 02:09:57.046988 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec2a56deaf64463aa75024fb5fa32f4" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d" is_pre_election: true
I20240502 02:09:57.047255 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec2a56deaf64463aa75024fb5fa32f4" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5956dd8d174df99a28e38e8a7a2524" is_pre_election: true
I20240502 02:09:57.047377 7268 raft_consensus.cc:2454] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 0.
I20240502 02:09:57.047490 7401 raft_consensus.cc:2454] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 0.
I20240502 02:09:57.047837 7468 leader_election.cc:302] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100889921252449596fc397b10fe903d, defc209b6ddf4b719d89059a990ecb05; no voters:
I20240502 02:09:57.048103 7870 raft_consensus.cc:2781] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:09:57.048179 7870 raft_consensus.cc:482] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:09:57.048220 7870 raft_consensus.cc:3037] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.049110 7870 raft_consensus.cc:504] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } }
I20240502 02:09:57.049455 7870 leader_election.cc:288] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 election: Requested vote from peers 100889921252449596fc397b10fe903d (127.27.111.193:43115), ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893)
I20240502 02:09:57.049830 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec2a56deaf64463aa75024fb5fa32f4" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5956dd8d174df99a28e38e8a7a2524"
I20240502 02:09:57.049831 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec2a56deaf64463aa75024fb5fa32f4" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d"
I20240502 02:09:57.049957 7401 raft_consensus.cc:3037] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.049958 7268 raft_consensus.cc:3037] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.051198 7401 raft_consensus.cc:2454] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 1.
I20240502 02:09:57.051508 7268 raft_consensus.cc:2454] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 1.
I20240502 02:09:57.051527 7466 leader_election.cc:302] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca5956dd8d174df99a28e38e8a7a2524, defc209b6ddf4b719d89059a990ecb05; no voters:
I20240502 02:09:57.051693 7870 raft_consensus.cc:2781] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:57.051949 7870 raft_consensus.cc:686] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Becoming Leader. State: Replica: defc209b6ddf4b719d89059a990ecb05, State: Running, Role: LEADER
I20240502 02:09:57.052268 7870 consensus_queue.cc:237] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } }
I20240502 02:09:57.055555 7123 catalog_manager.cc:5484] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 reported cstate change: term changed from 0 to 1, leader changed from <none> to defc209b6ddf4b719d89059a990ecb05 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: UNKNOWN } } }
W20240502 02:09:57.056855 7316 tablet.cc:2369] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 02:09:57.082279 7877 raft_consensus.cc:482] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:09:57.082502 7877 raft_consensus.cc:504] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.083582 7877 leader_election.cc:288] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 100889921252449596fc397b10fe903d (127.27.111.193:43115), defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:09:57.088012 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d090773bb34c6eb370f39ffae4e4ea" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d" is_pre_election: true
I20240502 02:09:57.088178 7268 raft_consensus.cc:2454] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5956dd8d174df99a28e38e8a7a2524 in term 0.
I20240502 02:09:57.088347 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d090773bb34c6eb370f39ffae4e4ea" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "defc209b6ddf4b719d89059a990ecb05" is_pre_election: true
I20240502 02:09:57.088559 7533 raft_consensus.cc:2454] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5956dd8d174df99a28e38e8a7a2524 in term 0.
I20240502 02:09:57.088532 7336 leader_election.cc:302] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100889921252449596fc397b10fe903d, ca5956dd8d174df99a28e38e8a7a2524; no voters:
I20240502 02:09:57.088783 7877 raft_consensus.cc:2781] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:09:57.088848 7877 raft_consensus.cc:482] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:09:57.088917 7877 raft_consensus.cc:3037] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.089896 7877 raft_consensus.cc:504] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.090272 7877 leader_election.cc:288] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 election: Requested vote from peers 100889921252449596fc397b10fe903d (127.27.111.193:43115), defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:09:57.090678 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d090773bb34c6eb370f39ffae4e4ea" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d"
I20240502 02:09:57.090663 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d090773bb34c6eb370f39ffae4e4ea" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "defc209b6ddf4b719d89059a990ecb05"
I20240502 02:09:57.090816 7533 raft_consensus.cc:3037] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.090816 7268 raft_consensus.cc:3037] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.091799 7268 raft_consensus.cc:2454] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5956dd8d174df99a28e38e8a7a2524 in term 1.
I20240502 02:09:57.091799 7533 raft_consensus.cc:2454] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5956dd8d174df99a28e38e8a7a2524 in term 1.
I20240502 02:09:57.092093 7335 leader_election.cc:302] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca5956dd8d174df99a28e38e8a7a2524, defc209b6ddf4b719d89059a990ecb05; no voters:
I20240502 02:09:57.092269 7877 raft_consensus.cc:2781] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:57.092514 7877 raft_consensus.cc:686] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Becoming Leader. State: Replica: ca5956dd8d174df99a28e38e8a7a2524, State: Running, Role: LEADER
I20240502 02:09:57.092885 7877 consensus_queue.cc:237] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.095953 7123 catalog_manager.cc:5484] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: UNKNOWN } } }
I20240502 02:09:57.190146 7877 raft_consensus.cc:482] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:09:57.190294 7877 raft_consensus.cc:504] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.190769 7877 leader_election.cc:288] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 100889921252449596fc397b10fe903d (127.27.111.193:43115), defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:09:57.191258 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b56f82e754429e8cc8fc6d2b78ce8e" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d" is_pre_election: true
I20240502 02:09:57.191274 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b56f82e754429e8cc8fc6d2b78ce8e" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "defc209b6ddf4b719d89059a990ecb05" is_pre_election: true
I20240502 02:09:57.191424 7533 raft_consensus.cc:2454] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5956dd8d174df99a28e38e8a7a2524 in term 0.
I20240502 02:09:57.191424 7268 raft_consensus.cc:2454] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5956dd8d174df99a28e38e8a7a2524 in term 0.
I20240502 02:09:57.191741 7335 leader_election.cc:302] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca5956dd8d174df99a28e38e8a7a2524, defc209b6ddf4b719d89059a990ecb05; no voters:
I20240502 02:09:57.191941 7877 raft_consensus.cc:2781] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:09:57.192006 7877 raft_consensus.cc:482] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:09:57.192034 7877 raft_consensus.cc:3037] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.192842 7877 raft_consensus.cc:504] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.193244 7877 leader_election.cc:288] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 election: Requested vote from peers 100889921252449596fc397b10fe903d (127.27.111.193:43115), defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:09:57.193647 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b56f82e754429e8cc8fc6d2b78ce8e" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d"
I20240502 02:09:57.193631 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b56f82e754429e8cc8fc6d2b78ce8e" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "defc209b6ddf4b719d89059a990ecb05"
I20240502 02:09:57.193758 7533 raft_consensus.cc:3037] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.193764 7268 raft_consensus.cc:3037] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.194541 7268 raft_consensus.cc:2454] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5956dd8d174df99a28e38e8a7a2524 in term 1.
I20240502 02:09:57.194648 7533 raft_consensus.cc:2454] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5956dd8d174df99a28e38e8a7a2524 in term 1.
I20240502 02:09:57.194844 7336 leader_election.cc:302] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100889921252449596fc397b10fe903d, ca5956dd8d174df99a28e38e8a7a2524; no voters:
I20240502 02:09:57.195022 7877 raft_consensus.cc:2781] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:57.195115 7877 raft_consensus.cc:686] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Becoming Leader. State: Replica: ca5956dd8d174df99a28e38e8a7a2524, State: Running, Role: LEADER
I20240502 02:09:57.195255 7877 consensus_queue.cc:237] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.196923 7123 catalog_manager.cc:5484] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: UNKNOWN } } }
W20240502 02:09:57.204219 7581 tablet.cc:2369] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 02:09:57.361644 7889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:57.361898 7889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:57.364898 7889 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 02:09:57.464190 7401 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:57.464298 7268 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:57.464959 7882 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:09:57.465101 7533 raft_consensus.cc:1261] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:57.465965 7877 consensus_queue.cc:1035] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:09:57.466050 7267 raft_consensus.cc:1261] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:57.467737 7870 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:09:57.471392 7877 consensus_queue.cc:1035] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:09:57.475561 7915 mvcc.cc:205] Tried to move back new op lower bound from 7023066306404773888 to 7023066304728838144. Current Snapshot: MvccSnapshot[applied={T|T < 7023066306404773888}]
I20240502 02:09:57.476128 7923 mvcc.cc:205] Tried to move back new op lower bound from 7023066306404773888 to 7023066304728838144. Current Snapshot: MvccSnapshot[applied={T|T < 7023066306404773888}]
I20240502 02:09:57.480667 7916 mvcc.cc:205] Tried to move back new op lower bound from 7023066306430316544 to 7023066305313042432. Current Snapshot: MvccSnapshot[applied={T|T < 7023066306410655744}]
I20240502 02:09:57.483930 7532 raft_consensus.cc:1261] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:57.484333 7886 consensus_queue.cc:1035] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:09:57.490022 7268 raft_consensus.cc:1261] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:57.490526 7886 consensus_queue.cc:1035] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 129.881 ms
time per row: 0.0649404 ms
I20240502 02:09:57.542006 7117 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55280:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 02:09:57.542527 7117 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 02:09:57.547434 7380 tablet_service.cc:1467] Processing CreateTablet for tablet c7c003a62d744c138c6ef685d826b7aa (DEFAULT_TABLE table=rebalance_test_table_1 [id=789c28d35c724bb8a67e0674bacf590b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:57.547766 7379 tablet_service.cc:1467] Processing CreateTablet for tablet 7287522a25cc459099d78147ca1ed1cd (DEFAULT_TABLE table=rebalance_test_table_1 [id=789c28d35c724bb8a67e0674bacf590b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:57.547782 7380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c003a62d744c138c6ef685d826b7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:57.548027 7379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7287522a25cc459099d78147ca1ed1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:57.548080 7381 tablet_service.cc:1467] Processing CreateTablet for tablet 4a048c9ba70c40e38c3dc38fd997c1bf (DEFAULT_TABLE table=rebalance_test_table_1 [id=789c28d35c724bb8a67e0674bacf590b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:57.548099 7247 tablet_service.cc:1467] Processing CreateTablet for tablet c7c003a62d744c138c6ef685d826b7aa (DEFAULT_TABLE table=rebalance_test_table_1 [id=789c28d35c724bb8a67e0674bacf590b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:57.548285 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a048c9ba70c40e38c3dc38fd997c1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:57.548374 7247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c003a62d744c138c6ef685d826b7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:57.550453 7940 tablet_bootstrap.cc:492] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d: Bootstrap starting.
I20240502 02:09:57.551138 7512 tablet_service.cc:1467] Processing CreateTablet for tablet c7c003a62d744c138c6ef685d826b7aa (DEFAULT_TABLE table=rebalance_test_table_1 [id=789c28d35c724bb8a67e0674bacf590b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:57.551239 7939 tablet_bootstrap.cc:492] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524: Bootstrap starting.
I20240502 02:09:57.551239 7511 tablet_service.cc:1467] Processing CreateTablet for tablet 7287522a25cc459099d78147ca1ed1cd (DEFAULT_TABLE table=rebalance_test_table_1 [id=789c28d35c724bb8a67e0674bacf590b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:57.551342 7940 tablet_bootstrap.cc:654] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:57.551440 7512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c003a62d744c138c6ef685d826b7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:57.551647 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 4a048c9ba70c40e38c3dc38fd997c1bf (DEFAULT_TABLE table=rebalance_test_table_1 [id=789c28d35c724bb8a67e0674bacf590b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:57.551853 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a048c9ba70c40e38c3dc38fd997c1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:57.552088 7939 tablet_bootstrap.cc:654] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:57.553567 7247 tablet_service.cc:1467] Processing CreateTablet for tablet 7287522a25cc459099d78147ca1ed1cd (DEFAULT_TABLE table=rebalance_test_table_1 [id=789c28d35c724bb8a67e0674bacf590b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:57.553658 7246 tablet_service.cc:1467] Processing CreateTablet for tablet 4a048c9ba70c40e38c3dc38fd997c1bf (DEFAULT_TABLE table=rebalance_test_table_1 [id=789c28d35c724bb8a67e0674bacf590b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:57.553782 7247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7287522a25cc459099d78147ca1ed1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:57.553939 7511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7287522a25cc459099d78147ca1ed1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:57.554134 7246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a048c9ba70c40e38c3dc38fd997c1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:57.555313 7940 tablet_bootstrap.cc:492] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d: No bootstrap required, opened a new log
I20240502 02:09:57.555405 7940 ts_tablet_manager.cc:1367] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 02:09:57.555866 7940 raft_consensus.cc:348] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.555953 7940 raft_consensus.cc:374] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:57.555979 7940 raft_consensus.cc:729] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100889921252449596fc397b10fe903d, State: Initialized, Role: FOLLOWER
I20240502 02:09:57.556109 7940 consensus_queue.cc:260] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.556322 7939 tablet_bootstrap.cc:492] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524: No bootstrap required, opened a new log
I20240502 02:09:57.556385 7940 ts_tablet_manager.cc:1398] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:09:57.556421 7939 ts_tablet_manager.cc:1367] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 02:09:57.556581 7940 tablet_bootstrap.cc:492] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d: Bootstrap starting.
I20240502 02:09:57.556866 7939 raft_consensus.cc:348] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.556978 7939 raft_consensus.cc:374] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:57.557013 7939 raft_consensus.cc:729] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5956dd8d174df99a28e38e8a7a2524, State: Initialized, Role: FOLLOWER
I20240502 02:09:57.557152 7939 consensus_queue.cc:260] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.557420 7939 ts_tablet_manager.cc:1398] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:09:57.557439 7940 tablet_bootstrap.cc:654] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:57.557600 7939 tablet_bootstrap.cc:492] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524: Bootstrap starting.
I20240502 02:09:57.558495 7939 tablet_bootstrap.cc:654] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:57.559119 7943 tablet_bootstrap.cc:492] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05: Bootstrap starting.
I20240502 02:09:57.559384 7940 tablet_bootstrap.cc:492] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d: No bootstrap required, opened a new log
I20240502 02:09:57.559459 7940 ts_tablet_manager.cc:1367] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 02:09:57.559867 7940 raft_consensus.cc:348] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.559949 7940 raft_consensus.cc:374] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:57.559980 7940 raft_consensus.cc:729] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100889921252449596fc397b10fe903d, State: Initialized, Role: FOLLOWER
I20240502 02:09:57.560058 7943 tablet_bootstrap.cc:654] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:57.560098 7940 consensus_queue.cc:260] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.560349 7940 ts_tablet_manager.cc:1398] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:09:57.560492 7940 tablet_bootstrap.cc:492] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d: Bootstrap starting.
I20240502 02:09:57.561079 7939 tablet_bootstrap.cc:492] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524: No bootstrap required, opened a new log
I20240502 02:09:57.561162 7939 ts_tablet_manager.cc:1367] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 02:09:57.561386 7940 tablet_bootstrap.cc:654] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:57.561641 7939 raft_consensus.cc:348] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.561758 7939 raft_consensus.cc:374] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:57.561800 7939 raft_consensus.cc:729] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5956dd8d174df99a28e38e8a7a2524, State: Initialized, Role: FOLLOWER
I20240502 02:09:57.561939 7939 consensus_queue.cc:260] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.562213 7939 ts_tablet_manager.cc:1398] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 02:09:57.562381 7939 tablet_bootstrap.cc:492] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: Bootstrap starting.
I20240502 02:09:57.562933 7940 tablet_bootstrap.cc:492] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d: No bootstrap required, opened a new log
I20240502 02:09:57.563066 7940 ts_tablet_manager.cc:1367] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 02:09:57.563350 7939 tablet_bootstrap.cc:654] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:57.563408 7886 raft_consensus.cc:482] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:09:57.563529 7886 raft_consensus.cc:504] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.563819 7940 raft_consensus.cc:348] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.563962 7940 raft_consensus.cc:374] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:57.563961 7886 leader_election.cc:288] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 100889921252449596fc397b10fe903d (127.27.111.193:43115), defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:09:57.564059 7940 raft_consensus.cc:729] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100889921252449596fc397b10fe903d, State: Initialized, Role: FOLLOWER
I20240502 02:09:57.564256 7943 tablet_bootstrap.cc:492] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05: No bootstrap required, opened a new log
I20240502 02:09:57.564220 7940 consensus_queue.cc:260] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.564348 7943 ts_tablet_manager.cc:1367] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 02:09:57.564450 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a048c9ba70c40e38c3dc38fd997c1bf" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "defc209b6ddf4b719d89059a990ecb05" is_pre_election: true
I20240502 02:09:57.564762 7940 ts_tablet_manager.cc:1398] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 02:09:57.564757 7943 raft_consensus.cc:348] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.564857 7943 raft_consensus.cc:374] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:57.564899 7943 raft_consensus.cc:729] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: defc209b6ddf4b719d89059a990ecb05, State: Initialized, Role: FOLLOWER
I20240502 02:09:57.564971 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a048c9ba70c40e38c3dc38fd997c1bf" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d" is_pre_election: true
I20240502 02:09:57.565062 7943 consensus_queue.cc:260] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.565094 7268 raft_consensus.cc:2454] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5956dd8d174df99a28e38e8a7a2524 in term 0.
I20240502 02:09:57.565361 7943 ts_tablet_manager.cc:1398] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:09:57.565421 7336 leader_election.cc:302] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100889921252449596fc397b10fe903d, ca5956dd8d174df99a28e38e8a7a2524; no voters:
W20240502 02:09:57.565577 7335 leader_election.cc:341] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407): Illegal state: must be running to vote when last-logged opid is not known
I20240502 02:09:57.565948 7943 tablet_bootstrap.cc:492] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05: Bootstrap starting.
I20240502 02:09:57.566038 7939 tablet_bootstrap.cc:492] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: No bootstrap required, opened a new log
I20240502 02:09:57.566128 7939 ts_tablet_manager.cc:1367] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 02:09:57.566443 7886 raft_consensus.cc:2781] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:09:57.566533 7886 raft_consensus.cc:482] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:09:57.566569 7939 raft_consensus.cc:348] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.566687 7939 raft_consensus.cc:374] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:57.566720 7939 raft_consensus.cc:729] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5956dd8d174df99a28e38e8a7a2524, State: Initialized, Role: FOLLOWER
I20240502 02:09:57.566569 7886 raft_consensus.cc:3037] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.566825 7939 consensus_queue.cc:260] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.567078 7939 ts_tablet_manager.cc:1398] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 02:09:57.567831 7886 raft_consensus.cc:504] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.568156 7943 tablet_bootstrap.cc:654] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:57.568226 7886 leader_election.cc:288] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 election: Requested vote from peers 100889921252449596fc397b10fe903d (127.27.111.193:43115), defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:09:57.568751 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a048c9ba70c40e38c3dc38fd997c1bf" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "defc209b6ddf4b719d89059a990ecb05"
I20240502 02:09:57.568799 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a048c9ba70c40e38c3dc38fd997c1bf" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d"
I20240502 02:09:57.568933 7268 raft_consensus.cc:3037] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.569728 7943 tablet_bootstrap.cc:492] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05: No bootstrap required, opened a new log
I20240502 02:09:57.569819 7943 ts_tablet_manager.cc:1367] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 02:09:57.569975 7268 raft_consensus.cc:2454] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5956dd8d174df99a28e38e8a7a2524 in term 1.
I20240502 02:09:57.570277 7336 leader_election.cc:302] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100889921252449596fc397b10fe903d, ca5956dd8d174df99a28e38e8a7a2524; no voters:
I20240502 02:09:57.570277 7943 raft_consensus.cc:348] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.570385 7943 raft_consensus.cc:374] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:57.570461 7943 raft_consensus.cc:729] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: defc209b6ddf4b719d89059a990ecb05, State: Initialized, Role: FOLLOWER
I20240502 02:09:57.570605 7943 consensus_queue.cc:260] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
W20240502 02:09:57.570650 7335 leader_election.cc:341] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407): Illegal state: must be running to vote when last-logged opid is not known
I20240502 02:09:57.570921 7943 ts_tablet_manager.cc:1398] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 02:09:57.571838 7943 tablet_bootstrap.cc:492] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05: Bootstrap starting.
I20240502 02:09:57.572788 7943 tablet_bootstrap.cc:654] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:57.573272 7886 raft_consensus.cc:2781] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:57.573400 7886 raft_consensus.cc:686] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Becoming Leader. State: Replica: ca5956dd8d174df99a28e38e8a7a2524, State: Running, Role: LEADER
I20240502 02:09:57.573535 7886 consensus_queue.cc:237] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.574054 7943 tablet_bootstrap.cc:492] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05: No bootstrap required, opened a new log
I20240502 02:09:57.574137 7943 ts_tablet_manager.cc:1367] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 02:09:57.574615 7943 raft_consensus.cc:348] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.574715 7943 raft_consensus.cc:374] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:57.574754 7943 raft_consensus.cc:729] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: defc209b6ddf4b719d89059a990ecb05, State: Initialized, Role: FOLLOWER
I20240502 02:09:57.574887 7943 consensus_queue.cc:260] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.575152 7943 ts_tablet_manager.cc:1398] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 02:09:57.575362 7123 catalog_manager.cc:5484] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: UNKNOWN } } }
I20240502 02:09:57.623744 7873 raft_consensus.cc:482] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:09:57.623879 7873 raft_consensus.cc:504] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.625065 7873 leader_election.cc:288] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893), defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:09:57.628513 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7287522a25cc459099d78147ca1ed1cd" candidate_uuid: "100889921252449596fc397b10fe903d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5956dd8d174df99a28e38e8a7a2524" is_pre_election: true
I20240502 02:09:57.628513 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7287522a25cc459099d78147ca1ed1cd" candidate_uuid: "100889921252449596fc397b10fe903d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "defc209b6ddf4b719d89059a990ecb05" is_pre_election: true
I20240502 02:09:57.628655 7401 raft_consensus.cc:2454] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100889921252449596fc397b10fe903d in term 0.
I20240502 02:09:57.628665 7533 raft_consensus.cc:2454] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100889921252449596fc397b10fe903d in term 0.
I20240502 02:09:57.628995 7201 leader_election.cc:302] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100889921252449596fc397b10fe903d, defc209b6ddf4b719d89059a990ecb05; no voters:
I20240502 02:09:57.629227 7873 raft_consensus.cc:2781] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:09:57.629284 7873 raft_consensus.cc:482] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:09:57.629310 7873 raft_consensus.cc:3037] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.630240 7873 raft_consensus.cc:504] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.630614 7873 leader_election.cc:288] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [CANDIDATE]: Term 1 election: Requested vote from peers ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893), defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:09:57.631049 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7287522a25cc459099d78147ca1ed1cd" candidate_uuid: "100889921252449596fc397b10fe903d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "defc209b6ddf4b719d89059a990ecb05"
I20240502 02:09:57.631177 7533 raft_consensus.cc:3037] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.632186 7533 raft_consensus.cc:2454] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100889921252449596fc397b10fe903d in term 1.
I20240502 02:09:57.634564 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7287522a25cc459099d78147ca1ed1cd" candidate_uuid: "100889921252449596fc397b10fe903d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5956dd8d174df99a28e38e8a7a2524"
I20240502 02:09:57.634701 7401 raft_consensus.cc:3037] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.635521 7401 raft_consensus.cc:2454] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100889921252449596fc397b10fe903d in term 1.
I20240502 02:09:57.635759 7201 leader_election.cc:302] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100889921252449596fc397b10fe903d, defc209b6ddf4b719d89059a990ecb05; no voters:
I20240502 02:09:57.638478 7873 raft_consensus.cc:2781] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:57.638679 7873 raft_consensus.cc:686] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [term 1 LEADER]: Becoming Leader. State: Replica: 100889921252449596fc397b10fe903d, State: Running, Role: LEADER
I20240502 02:09:57.638962 7873 consensus_queue.cc:237] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.644069 7123 catalog_manager.cc:5484] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d reported cstate change: term changed from 0 to 1, leader changed from <none> to 100889921252449596fc397b10fe903d (127.27.111.193). New cstate: current_term: 1 leader_uuid: "100889921252449596fc397b10fe903d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: UNKNOWN } } }
I20240502 02:09:57.815518 7882 raft_consensus.cc:482] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:09:57.815670 7882 raft_consensus.cc:504] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.816172 7882 leader_election.cc:288] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893), 100889921252449596fc397b10fe903d (127.27.111.193:43115)
I20240502 02:09:57.816673 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c003a62d744c138c6ef685d826b7aa" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5956dd8d174df99a28e38e8a7a2524" is_pre_election: true
I20240502 02:09:57.816675 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c003a62d744c138c6ef685d826b7aa" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d" is_pre_election: true
I20240502 02:09:57.816805 7401 raft_consensus.cc:2454] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 0.
I20240502 02:09:57.816820 7268 raft_consensus.cc:2454] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 0.
I20240502 02:09:57.817122 7468 leader_election.cc:302] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100889921252449596fc397b10fe903d, defc209b6ddf4b719d89059a990ecb05; no voters:
I20240502 02:09:57.817284 7882 raft_consensus.cc:2781] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:09:57.817343 7882 raft_consensus.cc:482] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:09:57.817384 7882 raft_consensus.cc:3037] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.818209 7882 raft_consensus.cc:504] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.818567 7882 leader_election.cc:288] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 election: Requested vote from peers ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893), 100889921252449596fc397b10fe903d (127.27.111.193:43115)
I20240502 02:09:57.818893 7268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c003a62d744c138c6ef685d826b7aa" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d"
I20240502 02:09:57.818909 7401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7c003a62d744c138c6ef685d826b7aa" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5956dd8d174df99a28e38e8a7a2524"
I20240502 02:09:57.818985 7268 raft_consensus.cc:3037] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.819020 7401 raft_consensus.cc:3037] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.819692 7268 raft_consensus.cc:2454] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 1.
I20240502 02:09:57.819783 7401 raft_consensus.cc:2454] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 1.
I20240502 02:09:57.819957 7468 leader_election.cc:302] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100889921252449596fc397b10fe903d, defc209b6ddf4b719d89059a990ecb05; no voters:
I20240502 02:09:57.820148 7882 raft_consensus.cc:2781] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:57.820255 7882 raft_consensus.cc:686] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Becoming Leader. State: Replica: defc209b6ddf4b719d89059a990ecb05, State: Running, Role: LEADER
I20240502 02:09:57.820446 7882 consensus_queue.cc:237] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:57.822371 7123 catalog_manager.cc:5484] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 reported cstate change: term changed from 0 to 1, leader changed from <none> to defc209b6ddf4b719d89059a990ecb05 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } }
W20240502 02:09:57.961061 7957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:57.961321 7957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:57.964591 7957 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 02:09:57.982272 7533 raft_consensus.cc:3037] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:57.983644 7886 consensus_queue.cc:1035] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 02:09:57.986801 7886 consensus_queue.cc:1035] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:09:58.053939 7268 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:58.053954 7401 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:58.054380 7882 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:09:58.054545 7870 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:09:58.061451 7401 raft_consensus.cc:1261] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer 100889921252449596fc397b10fe903d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:58.062031 7533 raft_consensus.cc:1261] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer 100889921252449596fc397b10fe903d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:58.062096 7954 consensus_queue.cc:1035] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:09:58.068289 7954 consensus_queue.cc:1035] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.836 ms
time per row: 0.0559181 ms
I20240502 02:09:58.121580 7120 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55280:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 02:09:58.122159 7120 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 02:09:58.127660 7246 tablet_service.cc:1467] Processing CreateTablet for tablet e6b4b64d18f34c0298369e247e670fcd (DEFAULT_TABLE table=rebalance_test_table_2 [id=178c22953cac45a28113df2575de7ad7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:58.127738 7381 tablet_service.cc:1467] Processing CreateTablet for tablet e6b4b64d18f34c0298369e247e670fcd (DEFAULT_TABLE table=rebalance_test_table_2 [id=178c22953cac45a28113df2575de7ad7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:58.127856 7511 tablet_service.cc:1467] Processing CreateTablet for tablet e6b4b64d18f34c0298369e247e670fcd (DEFAULT_TABLE table=rebalance_test_table_2 [id=178c22953cac45a28113df2575de7ad7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:58.127995 7246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b4b64d18f34c0298369e247e670fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:58.127995 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b4b64d18f34c0298369e247e670fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:58.128154 7511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b4b64d18f34c0298369e247e670fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:58.129917 8014 tablet_bootstrap.cc:492] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05: Bootstrap starting.
I20240502 02:09:58.130450 8015 tablet_bootstrap.cc:492] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d: Bootstrap starting.
I20240502 02:09:58.130899 8014 tablet_bootstrap.cc:654] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:58.131112 7246 tablet_service.cc:1467] Processing CreateTablet for tablet 00e19846903d4a088e91ba3efaaa6305 (DEFAULT_TABLE table=rebalance_test_table_2 [id=178c22953cac45a28113df2575de7ad7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:58.131115 7380 tablet_service.cc:1467] Processing CreateTablet for tablet 4347d62ce08b4f7fa32f48fb769bba68 (DEFAULT_TABLE table=rebalance_test_table_2 [id=178c22953cac45a28113df2575de7ad7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:58.131297 8015 tablet_bootstrap.cc:654] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:58.131340 7246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e19846903d4a088e91ba3efaaa6305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:58.131363 7380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4347d62ce08b4f7fa32f48fb769bba68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:58.132390 8015 tablet_bootstrap.cc:492] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d: No bootstrap required, opened a new log
I20240502 02:09:58.132470 8015 ts_tablet_manager.cc:1367] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 02:09:58.132908 8015 raft_consensus.cc:348] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.133013 8015 raft_consensus.cc:374] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:58.132983 7247 tablet_service.cc:1467] Processing CreateTablet for tablet 4347d62ce08b4f7fa32f48fb769bba68 (DEFAULT_TABLE table=rebalance_test_table_2 [id=178c22953cac45a28113df2575de7ad7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:58.133045 8015 raft_consensus.cc:729] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100889921252449596fc397b10fe903d, State: Initialized, Role: FOLLOWER
I20240502 02:09:58.133172 7247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4347d62ce08b4f7fa32f48fb769bba68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:58.133183 8015 consensus_queue.cc:260] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.133564 8015 ts_tablet_manager.cc:1398] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:09:58.133761 8015 tablet_bootstrap.cc:492] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d: Bootstrap starting.
I20240502 02:09:58.133911 7381 tablet_service.cc:1467] Processing CreateTablet for tablet 00e19846903d4a088e91ba3efaaa6305 (DEFAULT_TABLE table=rebalance_test_table_2 [id=178c22953cac45a28113df2575de7ad7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:58.134148 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e19846903d4a088e91ba3efaaa6305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:58.134639 8015 tablet_bootstrap.cc:654] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:58.135969 8015 tablet_bootstrap.cc:492] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d: No bootstrap required, opened a new log
I20240502 02:09:58.136063 8015 ts_tablet_manager.cc:1367] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:58.136440 8016 tablet_bootstrap.cc:492] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524: Bootstrap starting.
I20240502 02:09:58.136504 8015 raft_consensus.cc:348] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.136600 8015 raft_consensus.cc:374] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:58.136554 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 4347d62ce08b4f7fa32f48fb769bba68 (DEFAULT_TABLE table=rebalance_test_table_2 [id=178c22953cac45a28113df2575de7ad7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:58.136631 8015 raft_consensus.cc:729] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100889921252449596fc397b10fe903d, State: Initialized, Role: FOLLOWER
I20240502 02:09:58.136744 8015 consensus_queue.cc:260] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.136798 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4347d62ce08b4f7fa32f48fb769bba68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:58.136997 8015 ts_tablet_manager.cc:1398] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:09:58.137079 8016 tablet_bootstrap.cc:654] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:58.137176 8015 tablet_bootstrap.cc:492] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d: Bootstrap starting.
I20240502 02:09:58.137996 8016 tablet_bootstrap.cc:492] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524: No bootstrap required, opened a new log
I20240502 02:09:58.137988 8015 tablet_bootstrap.cc:654] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:58.138067 8016 ts_tablet_manager.cc:1367] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 02:09:58.138765 8016 raft_consensus.cc:348] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.138881 8016 raft_consensus.cc:374] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:58.138921 8016 raft_consensus.cc:729] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5956dd8d174df99a28e38e8a7a2524, State: Initialized, Role: FOLLOWER
I20240502 02:09:58.139076 8016 consensus_queue.cc:260] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.139464 8016 ts_tablet_manager.cc:1398] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 02:09:58.139474 7511 tablet_service.cc:1467] Processing CreateTablet for tablet 00e19846903d4a088e91ba3efaaa6305 (DEFAULT_TABLE table=rebalance_test_table_2 [id=178c22953cac45a28113df2575de7ad7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:09:58.139714 7511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e19846903d4a088e91ba3efaaa6305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:09:58.139760 8016 tablet_bootstrap.cc:492] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524: Bootstrap starting.
I20240502 02:09:58.140524 8016 tablet_bootstrap.cc:654] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:58.141691 8015 tablet_bootstrap.cc:492] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d: No bootstrap required, opened a new log
I20240502 02:09:58.141786 8015 ts_tablet_manager.cc:1367] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 02:09:58.142284 8015 raft_consensus.cc:348] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.142412 8015 raft_consensus.cc:374] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:58.142478 8015 raft_consensus.cc:729] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100889921252449596fc397b10fe903d, State: Initialized, Role: FOLLOWER
I20240502 02:09:58.142633 8015 consensus_queue.cc:260] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.142899 8015 ts_tablet_manager.cc:1398] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:09:58.143040 8014 tablet_bootstrap.cc:492] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05: No bootstrap required, opened a new log
I20240502 02:09:58.143137 8014 ts_tablet_manager.cc:1367] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 02:09:58.143582 8016 tablet_bootstrap.cc:492] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524: No bootstrap required, opened a new log
I20240502 02:09:58.143675 8016 ts_tablet_manager.cc:1367] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 02:09:58.143795 8014 raft_consensus.cc:348] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.143906 8014 raft_consensus.cc:374] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:58.143939 8014 raft_consensus.cc:729] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: defc209b6ddf4b719d89059a990ecb05, State: Initialized, Role: FOLLOWER
I20240502 02:09:58.144068 8014 consensus_queue.cc:260] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.144115 8016 raft_consensus.cc:348] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.144213 8016 raft_consensus.cc:374] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:58.144243 8016 raft_consensus.cc:729] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5956dd8d174df99a28e38e8a7a2524, State: Initialized, Role: FOLLOWER
I20240502 02:09:58.144328 8014 ts_tablet_manager.cc:1398] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:09:58.144371 8016 consensus_queue.cc:260] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.144495 8014 tablet_bootstrap.cc:492] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05: Bootstrap starting.
I20240502 02:09:58.144632 8016 ts_tablet_manager.cc:1398] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:09:58.144954 8016 tablet_bootstrap.cc:492] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524: Bootstrap starting.
I20240502 02:09:58.145401 8014 tablet_bootstrap.cc:654] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:58.145790 8016 tablet_bootstrap.cc:654] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:58.146528 8014 tablet_bootstrap.cc:492] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05: No bootstrap required, opened a new log
I20240502 02:09:58.146610 8014 ts_tablet_manager.cc:1367] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 02:09:58.146915 8016 tablet_bootstrap.cc:492] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524: No bootstrap required, opened a new log
I20240502 02:09:58.146996 8016 ts_tablet_manager.cc:1367] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 02:09:58.147037 8014 raft_consensus.cc:348] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.147147 8014 raft_consensus.cc:374] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:58.147186 8014 raft_consensus.cc:729] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: defc209b6ddf4b719d89059a990ecb05, State: Initialized, Role: FOLLOWER
I20240502 02:09:58.147300 8014 consensus_queue.cc:260] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.147449 8016 raft_consensus.cc:348] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.147542 8016 raft_consensus.cc:374] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:58.147574 8016 raft_consensus.cc:729] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5956dd8d174df99a28e38e8a7a2524, State: Initialized, Role: FOLLOWER
I20240502 02:09:58.147661 8014 ts_tablet_manager.cc:1398] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 02:09:58.147711 8016 consensus_queue.cc:260] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.147810 8014 tablet_bootstrap.cc:492] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05: Bootstrap starting.
I20240502 02:09:58.147954 8016 ts_tablet_manager.cc:1398] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 02:09:58.148552 8014 tablet_bootstrap.cc:654] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05: Neither blocks nor log segments found. Creating new log.
I20240502 02:09:58.150185 8014 tablet_bootstrap.cc:492] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05: No bootstrap required, opened a new log
I20240502 02:09:58.150274 8014 ts_tablet_manager.cc:1367] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 02:09:58.150671 8014 raft_consensus.cc:348] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.150756 8014 raft_consensus.cc:374] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:09:58.150777 8014 raft_consensus.cc:729] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: defc209b6ddf4b719d89059a990ecb05, State: Initialized, Role: FOLLOWER
I20240502 02:09:58.150887 8014 consensus_queue.cc:260] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.151142 8014 ts_tablet_manager.cc:1398] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 02:09:58.194788 7877 raft_consensus.cc:482] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:09:58.194967 7877 raft_consensus.cc:504] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.195446 7877 leader_election.cc:288] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 100889921252449596fc397b10fe903d (127.27.111.193:43115), defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:09:58.195858 7267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4347d62ce08b4f7fa32f48fb769bba68" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d" is_pre_election: true
I20240502 02:09:58.195936 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4347d62ce08b4f7fa32f48fb769bba68" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "defc209b6ddf4b719d89059a990ecb05" is_pre_election: true
I20240502 02:09:58.196031 7267 raft_consensus.cc:2454] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5956dd8d174df99a28e38e8a7a2524 in term 0.
I20240502 02:09:58.196080 7533 raft_consensus.cc:2454] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5956dd8d174df99a28e38e8a7a2524 in term 0.
I20240502 02:09:58.196393 7336 leader_election.cc:302] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100889921252449596fc397b10fe903d, ca5956dd8d174df99a28e38e8a7a2524; no voters:
I20240502 02:09:58.196574 7877 raft_consensus.cc:2781] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:09:58.196642 7877 raft_consensus.cc:482] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:09:58.196669 7877 raft_consensus.cc:3037] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:58.197549 7877 raft_consensus.cc:504] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.197883 7877 leader_election.cc:288] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 election: Requested vote from peers 100889921252449596fc397b10fe903d (127.27.111.193:43115), defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:09:58.198230 7267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4347d62ce08b4f7fa32f48fb769bba68" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d"
I20240502 02:09:58.198230 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4347d62ce08b4f7fa32f48fb769bba68" candidate_uuid: "ca5956dd8d174df99a28e38e8a7a2524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "defc209b6ddf4b719d89059a990ecb05"
I20240502 02:09:58.198351 7533 raft_consensus.cc:3037] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:58.198351 7267 raft_consensus.cc:3037] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:58.199363 7267 raft_consensus.cc:2454] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5956dd8d174df99a28e38e8a7a2524 in term 1.
I20240502 02:09:58.199363 7533 raft_consensus.cc:2454] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5956dd8d174df99a28e38e8a7a2524 in term 1.
I20240502 02:09:58.199656 7335 leader_election.cc:302] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca5956dd8d174df99a28e38e8a7a2524, defc209b6ddf4b719d89059a990ecb05; no voters:
I20240502 02:09:58.199877 7877 raft_consensus.cc:2781] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:58.199970 7877 raft_consensus.cc:686] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Becoming Leader. State: Replica: ca5956dd8d174df99a28e38e8a7a2524, State: Running, Role: LEADER
I20240502 02:09:58.200124 7877 consensus_queue.cc:237] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.201887 7123 catalog_manager.cc:5484] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: UNKNOWN } } }
I20240502 02:09:58.229488 7882 raft_consensus.cc:482] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:09:58.229638 7882 raft_consensus.cc:504] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.230093 7882 leader_election.cc:288] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 100889921252449596fc397b10fe903d (127.27.111.193:43115), ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893)
I20240502 02:09:58.230639 7400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e19846903d4a088e91ba3efaaa6305" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5956dd8d174df99a28e38e8a7a2524" is_pre_election: true
I20240502 02:09:58.230612 7267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e19846903d4a088e91ba3efaaa6305" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d" is_pre_election: true
I20240502 02:09:58.230785 7267 raft_consensus.cc:2454] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 0.
I20240502 02:09:58.230784 7400 raft_consensus.cc:2454] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 0.
I20240502 02:09:58.231103 7468 leader_election.cc:302] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100889921252449596fc397b10fe903d, defc209b6ddf4b719d89059a990ecb05; no voters:
I20240502 02:09:58.231266 7882 raft_consensus.cc:2781] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:09:58.231326 7882 raft_consensus.cc:482] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:09:58.231353 7882 raft_consensus.cc:3037] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:58.232189 7882 raft_consensus.cc:504] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.232475 7882 leader_election.cc:288] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 election: Requested vote from peers 100889921252449596fc397b10fe903d (127.27.111.193:43115), ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893)
I20240502 02:09:58.232808 7267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e19846903d4a088e91ba3efaaa6305" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d"
I20240502 02:09:58.232834 7400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00e19846903d4a088e91ba3efaaa6305" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5956dd8d174df99a28e38e8a7a2524"
I20240502 02:09:58.232913 7267 raft_consensus.cc:3037] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:58.232961 7400 raft_consensus.cc:3037] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:58.233667 7267 raft_consensus.cc:2454] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 1.
I20240502 02:09:58.233798 7400 raft_consensus.cc:2454] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 1.
I20240502 02:09:58.233964 7468 leader_election.cc:302] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100889921252449596fc397b10fe903d, defc209b6ddf4b719d89059a990ecb05; no voters:
I20240502 02:09:58.234135 7882 raft_consensus.cc:2781] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:58.234265 7882 raft_consensus.cc:686] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Becoming Leader. State: Replica: defc209b6ddf4b719d89059a990ecb05, State: Running, Role: LEADER
I20240502 02:09:58.234452 7882 consensus_queue.cc:237] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.236028 7123 catalog_manager.cc:5484] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 reported cstate change: term changed from 0 to 1, leader changed from <none> to defc209b6ddf4b719d89059a990ecb05 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } }
I20240502 02:09:58.384042 7882 raft_consensus.cc:482] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:09:58.384186 7882 raft_consensus.cc:504] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.384599 7882 leader_election.cc:288] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 100889921252449596fc397b10fe903d (127.27.111.193:43115), ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893)
I20240502 02:09:58.385110 7400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b4b64d18f34c0298369e247e670fcd" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5956dd8d174df99a28e38e8a7a2524" is_pre_election: true
I20240502 02:09:58.385110 7267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b4b64d18f34c0298369e247e670fcd" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d" is_pre_election: true
I20240502 02:09:58.385267 7267 raft_consensus.cc:2454] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 0.
I20240502 02:09:58.385265 7400 raft_consensus.cc:2454] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 0.
I20240502 02:09:58.385571 7468 leader_election.cc:302] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100889921252449596fc397b10fe903d, defc209b6ddf4b719d89059a990ecb05; no voters:
I20240502 02:09:58.385749 7882 raft_consensus.cc:2781] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:09:58.385848 7882 raft_consensus.cc:482] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:09:58.385903 7882 raft_consensus.cc:3037] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:58.386780 7882 raft_consensus.cc:504] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.387123 7882 leader_election.cc:288] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 election: Requested vote from peers 100889921252449596fc397b10fe903d (127.27.111.193:43115), ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893)
I20240502 02:09:58.387498 7400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b4b64d18f34c0298369e247e670fcd" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5956dd8d174df99a28e38e8a7a2524"
I20240502 02:09:58.387498 7267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b4b64d18f34c0298369e247e670fcd" candidate_uuid: "defc209b6ddf4b719d89059a990ecb05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100889921252449596fc397b10fe903d"
I20240502 02:09:58.387648 7267 raft_consensus.cc:3037] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:58.387646 7400 raft_consensus.cc:3037] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:09:58.388636 7267 raft_consensus.cc:2454] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 1.
I20240502 02:09:58.388636 7400 raft_consensus.cc:2454] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate defc209b6ddf4b719d89059a990ecb05 in term 1.
I20240502 02:09:58.388973 7466 leader_election.cc:302] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca5956dd8d174df99a28e38e8a7a2524, defc209b6ddf4b719d89059a990ecb05; no voters:
I20240502 02:09:58.389171 7882 raft_consensus.cc:2781] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:09:58.389259 7882 raft_consensus.cc:686] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Becoming Leader. State: Replica: defc209b6ddf4b719d89059a990ecb05, State: Running, Role: LEADER
I20240502 02:09:58.389402 7882 consensus_queue.cc:237] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } }
I20240502 02:09:58.391103 7123 catalog_manager.cc:5484] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 reported cstate change: term changed from 0 to 1, leader changed from <none> to defc209b6ddf4b719d89059a990ecb05 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } }
W20240502 02:09:58.538914 8029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:58.539162 8029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:58.542484 8029 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 02:09:58.621659 7533 raft_consensus.cc:1261] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:58.621659 7267 raft_consensus.cc:1261] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:58.621670 7268 raft_consensus.cc:1261] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:58.622023 7400 raft_consensus.cc:1261] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:58.622385 7886 consensus_queue.cc:1035] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:09:58.622423 7870 consensus_queue.cc:1035] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:09:58.622534 7877 consensus_queue.cc:1035] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:09:58.622699 7870 consensus_queue.cc:1035] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:09:58.623826 7266 raft_consensus.cc:1261] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:58.624190 7870 consensus_queue.cc:1035] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:09:58.626534 7401 raft_consensus.cc:1261] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:09:58.627710 7882 consensus_queue.cc:1035] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.468 ms
time per row: 0.0532342 ms
I20240502 02:09:58.689324 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.197:36289
--local_ip_for_outbound_sockets=127.27.111.197
--tserver_master_addrs=127.27.111.254:42347
--webserver_port=37183
--webserver_interface=127.27.111.197
--builtin_ntp_servers=127.27.111.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 02:09:58.805967 8086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:58.806223 8086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:58.806295 8086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:58.809453 8086 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:09:58.809564 8086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:58.809669 8086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.197
I20240502 02:09:58.813339 8086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.197:36289
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.27.111.197
--webserver_port=37183
--tserver_master_addrs=127.27.111.254:42347
--never_fsync=true
--heap_profile_path=/tmp/kudu.8086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.197
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:58.814365 8086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:58.815575 8086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:58.823912 8092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:58.824530 8091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:58.825160 8094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:58.825891 8086 server_base.cc:1034] running on GCE node
I20240502 02:09:58.826316 8086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:58.827045 8086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:58.828208 8086 hybrid_clock.cc:648] HybridClock initialized: now 1714615798828196 us; error 28 us; skew 500 ppm
I20240502 02:09:58.830036 8086 webserver.cc:460] Webserver started at http://127.27.111.197:37183/ using document root <none> and password file <none>
I20240502 02:09:58.830687 8086 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:58.830763 8086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:58.834116 8086 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 02:09:58.836076 8100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:58.837090 8086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 02:09:58.837194 8086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "85e53716ea13433289fe6435b6693a5b"
format_stamp: "Formatted at 2024-05-02 02:09:53 on dist-test-slave-0pw1"
I20240502 02:09:58.837666 8086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:58.850869 8086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:58.851487 8086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:58.851640 8086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:58.852162 8086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:09:58.853072 8086 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:09:58.853121 8086 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:58.853168 8086 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:09:58.853195 8086 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:58.861569 8086 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.197:36289
I20240502 02:09:58.861590 8213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.197:36289 every 8 connection(s)
I20240502 02:09:58.862686 8086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 02:09:58.864775 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 8086
I20240502 02:09:58.865160 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.196:45431
--local_ip_for_outbound_sockets=127.27.111.196
--tserver_master_addrs=127.27.111.254:42347
--webserver_port=33443
--webserver_interface=127.27.111.196
--builtin_ntp_servers=127.27.111.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:09:58.872153 8214 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42347
I20240502 02:09:58.872365 8214 heartbeater.cc:464] Registering TS with master...
I20240502 02:09:58.872862 8214 heartbeater.cc:511] Master 127.27.111.254:42347 requested a full tablet report, sending...
I20240502 02:09:58.874193 7122 ts_manager.cc:194] Re-registered known tserver with Master: 85e53716ea13433289fe6435b6693a5b (127.27.111.197:36289)
I20240502 02:09:58.875056 7122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.197:40137
W20240502 02:09:58.976011 8217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:09:58.976263 8217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:09:58.976313 8217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:09:58.979451 8217 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:09:58.979518 8217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:09:58.979583 8217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.196
I20240502 02:09:58.983249 8217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.196:45431
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.111.196
--webserver_port=33443
--tserver_master_addrs=127.27.111.254:42347
--never_fsync=true
--heap_profile_path=/tmp/kudu.8217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.196
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:09:58.984161 8217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:09:58.985276 8217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:09:58.993661 8222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:58.994156 8225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:09:58.994906 8223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:09:58.995504 8217 server_base.cc:1034] running on GCE node
I20240502 02:09:58.995936 8217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:09:58.996688 8217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:09:58.997851 8217 hybrid_clock.cc:648] HybridClock initialized: now 1714615798997841 us; error 30 us; skew 500 ppm
I20240502 02:09:58.999814 8217 webserver.cc:460] Webserver started at http://127.27.111.196:33443/ using document root <none> and password file <none>
I20240502 02:09:59.000396 8217 fs_manager.cc:359] Metadata directory not provided
I20240502 02:09:59.000500 8217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:09:59.004078 8217 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 02:09:59.006145 8231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:59.007349 8217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 02:09:59.007501 8217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "efabf88707da4c13b7ec85bbfd8a89da"
format_stamp: "Formatted at 2024-05-02 02:09:52 on dist-test-slave-0pw1"
I20240502 02:09:59.007930 8217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:09:59.032640 8217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:09:59.033243 8217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:09:59.033440 8217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:09:59.033979 8217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:09:59.034916 8217 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:09:59.034973 8217 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:59.035032 8217 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:09:59.035063 8217 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:09:59.044482 8217 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.196:45431
I20240502 02:09:59.044505 8344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.196:45431 every 8 connection(s)
I20240502 02:09:59.045536 8217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 02:09:59.052884 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 8217
I20240502 02:09:59.054358 8345 heartbeater.cc:347] Connected to a master server at 127.27.111.254:42347
I20240502 02:09:59.054663 8345 heartbeater.cc:464] Registering TS with master...
I20240502 02:09:59.055233 8345 heartbeater.cc:511] Master 127.27.111.254:42347 requested a full tablet report, sending...
I20240502 02:09:59.056325 7122 ts_manager.cc:194] Re-registered known tserver with Master: efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196:45431)
I20240502 02:09:59.057097 7122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.196:58469
I20240502 02:09:59.247082 8148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:59.248380 7511 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:09:59.248564 7247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:59.249859 7381 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:09:59.270079 8279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:59.361939 8148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:59.362746 7247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:59.368170 7381 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:09:59.375344 8279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:59.384922 7511 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:09:59.480091 8279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:59.480940 8148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:59.481343 7247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:59.490082 7381 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:09:59.492138 7511 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:09:59.575309 8148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:59.576148 8279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:09:59.579641 7511 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:09:59.580962 7247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:09:59.591876 7381 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:09:59.625952 7533 consensus_queue.cc:237] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
I20240502 02:09:59.627511 7268 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:09:59.627619 7401 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:09:59.627986 8493 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:09:59.628171 8492 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:09:59.630270 8493 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
I20240502 02:09:59.630828 7268 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
W20240502 02:09:59.631537 7469 consensus_peers.cc:485] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 -> Peer 85e53716ea13433289fe6435b6693a5b (127.27.111.197:36289): Couldn't send request to peer 85e53716ea13433289fe6435b6693a5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ec2a56deaf64463aa75024fb5fa32f4. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:59.632337 7401 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
I20240502 02:09:59.633911 7116 catalog_manager.cc:5484] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 reported cstate change: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:09:59.641912 7533 consensus_queue.cc:237] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } }
I20240502 02:09:59.643594 7401 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:09:59.644012 7268 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:09:59.644029 7870 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:09:59.645588 7870 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:09:59.646430 8493 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } } }
W20240502 02:09:59.646948 7467 consensus_peers.cc:485] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 -> Peer efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196:45431): Couldn't send request to peer efabf88707da4c13b7ec85bbfd8a89da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7c003a62d744c138c6ef685d826b7aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:59.647070 7401 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } } }
I20240502 02:09:59.648000 7268 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } } }
I20240502 02:09:59.649700 7120 catalog_manager.cc:5484] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 reported cstate change: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:09:59.657363 7401 consensus_queue.cc:237] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
I20240502 02:09:59.658823 7268 raft_consensus.cc:1261] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:09:59.659138 7533 raft_consensus.cc:1261] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:09:59.659276 8518 consensus_queue.cc:1035] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:09:59.659806 8517 consensus_queue.cc:1035] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:09:59.660766 8518 raft_consensus.cc:2932] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
I20240502 02:09:59.661176 7268 raft_consensus.cc:2932] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
I20240502 02:09:59.661721 7533 raft_consensus.cc:2932] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
I20240502 02:09:59.664098 7116 catalog_manager.cc:5484] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d reported cstate change: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
W20240502 02:09:59.665441 7337 consensus_peers.cc:485] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 -> Peer 85e53716ea13433289fe6435b6693a5b (127.27.111.197:36289): Couldn't send request to peer 85e53716ea13433289fe6435b6693a5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4347d62ce08b4f7fa32f48fb769bba68. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:59.671099 7533 consensus_queue.cc:237] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } }
I20240502 02:09:59.672405 7268 raft_consensus.cc:1261] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:09:59.672525 7401 raft_consensus.cc:1261] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 02:09:59.672653 7467 consensus_peers.cc:485] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 -> Peer efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196:45431): Couldn't send request to peer efabf88707da4c13b7ec85bbfd8a89da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6b4b64d18f34c0298369e247e670fcd. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:59.672863 8498 consensus_queue.cc:1035] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:09:59.673008 8493 consensus_queue.cc:1035] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:09:59.674574 8492 raft_consensus.cc:2932] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } } }
I20240502 02:09:59.674947 7401 raft_consensus.cc:2932] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } } }
I20240502 02:09:59.676492 7268 raft_consensus.cc:2932] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } } }
I20240502 02:09:59.676983 7122 catalog_manager.cc:5484] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } } }
I20240502 02:09:59.684713 7268 consensus_queue.cc:237] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
I20240502 02:09:59.686143 7401 raft_consensus.cc:1261] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer 100889921252449596fc397b10fe903d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:09:59.686532 7533 raft_consensus.cc:1261] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer 100889921252449596fc397b10fe903d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:09:59.686560 8011 consensus_queue.cc:1035] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:09:59.686968 8535 consensus_queue.cc:1035] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:09:59.688263 8534 raft_consensus.cc:2932] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
I20240502 02:09:59.688613 7401 raft_consensus.cc:2932] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
I20240502 02:09:59.689057 7533 raft_consensus.cc:2932] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
I20240502 02:09:59.691001 7122 catalog_manager.cc:5484] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "100889921252449596fc397b10fe903d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
W20240502 02:09:59.691951 7203 consensus_peers.cc:485] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d -> Peer 85e53716ea13433289fe6435b6693a5b (127.27.111.197:36289): Couldn't send request to peer 85e53716ea13433289fe6435b6693a5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7287522a25cc459099d78147ca1ed1cd. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:59.698117 7401 consensus_queue.cc:237] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } }
I20240502 02:09:59.703096 7533 raft_consensus.cc:1261] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:09:59.703096 7268 raft_consensus.cc:1261] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:09:59.703512 8078 consensus_queue.cc:1035] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:09:59.703716 8520 consensus_queue.cc:1035] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:09:59.705513 8520 raft_consensus.cc:2932] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } } }
I20240502 02:09:59.705897 7533 raft_consensus.cc:2932] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } } }
I20240502 02:09:59.707209 7266 raft_consensus.cc:2932] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } } }
I20240502 02:09:59.708360 7116 catalog_manager.cc:5484] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 02:09:59.708750 7335 consensus_peers.cc:485] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 -> Peer efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196:45431): Couldn't send request to peer efabf88707da4c13b7ec85bbfd8a89da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7d090773bb34c6eb370f39ffae4e4ea. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:59.719318 7401 consensus_queue.cc:237] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
W20240502 02:09:59.720921 7337 consensus_peers.cc:485] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 -> Peer 85e53716ea13433289fe6435b6693a5b (127.27.111.197:36289): Couldn't send request to peer 85e53716ea13433289fe6435b6693a5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7b56f82e754429e8cc8fc6d2b78ce8e. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:09:59.720921 7266 raft_consensus.cc:1261] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:09:59.721136 7533 raft_consensus.cc:1261] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:09:59.721324 8518 consensus_queue.cc:1035] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:09:59.721524 8078 consensus_queue.cc:1035] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:09:59.723193 8517 raft_consensus.cc:2932] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
I20240502 02:09:59.723572 7533 raft_consensus.cc:2932] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
I20240502 02:09:59.723605 7266 raft_consensus.cc:2932] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
I20240502 02:09:59.725698 7122 catalog_manager.cc:5484] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:09:59.877110 8214 heartbeater.cc:503] Master 127.27.111.254:42347 was elected leader, sending a full tablet report...
I20240502 02:10:00.056723 8625 ts_tablet_manager.cc:902] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b: Initiating tablet copy from peer ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893)
I20240502 02:10:00.057401 8625 tablet_copy_client.cc:308] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:40893
I20240502 02:10:00.059742 8345 heartbeater.cc:503] Master 127.27.111.254:42347 was elected leader, sending a full tablet report...
I20240502 02:10:00.062513 7421 tablet_copy_service.cc:140] P ca5956dd8d174df99a28e38e8a7a2524: Received BeginTabletCopySession request for tablet 4347d62ce08b4f7fa32f48fb769bba68 from peer 85e53716ea13433289fe6435b6693a5b ({username='slave'} at 127.27.111.197:42227)
I20240502 02:10:00.062683 7421 tablet_copy_service.cc:161] P ca5956dd8d174df99a28e38e8a7a2524: Beginning new tablet copy session on tablet 4347d62ce08b4f7fa32f48fb769bba68 from peer 85e53716ea13433289fe6435b6693a5b at {username='slave'} at 127.27.111.197:42227: session id = 85e53716ea13433289fe6435b6693a5b-4347d62ce08b4f7fa32f48fb769bba68
I20240502 02:10:00.063832 7421 tablet_copy_source_session.cc:205] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:10:00.067034 8625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4347d62ce08b4f7fa32f48fb769bba68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:00.072541 8625 tablet_copy_client.cc:783] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b: tablet copy: Starting download of 0 data blocks...
I20240502 02:10:00.072758 8625 tablet_copy_client.cc:647] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b: tablet copy: Starting download of 1 WAL segments...
I20240502 02:10:00.074357 8625 tablet_copy_client.cc:523] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:10:00.075937 8625 tablet_bootstrap.cc:492] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b: Bootstrap starting.
I20240502 02:10:00.080948 8625 log.cc:824] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b: Log is configured to *not* fsync() on all Append() calls
I20240502 02:10:00.099813 8625 tablet_bootstrap.cc:492] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:10:00.100183 8625 tablet_bootstrap.cc:492] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b: Bootstrap complete.
I20240502 02:10:00.100474 8625 ts_tablet_manager.cc:1367] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b: Time spent bootstrapping tablet: real 0.025s user 0.024s sys 0.000s
I20240502 02:10:00.102463 8625 raft_consensus.cc:348] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
I20240502 02:10:00.102707 8625 raft_consensus.cc:729] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85e53716ea13433289fe6435b6693a5b, State: Initialized, Role: LEARNER
I20240502 02:10:00.102998 8625 consensus_queue.cc:260] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
I20240502 02:10:00.103832 8625 ts_tablet_manager.cc:1398] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 02:10:00.104588 7421 tablet_copy_service.cc:342] P ca5956dd8d174df99a28e38e8a7a2524: Request end of tablet copy session 85e53716ea13433289fe6435b6693a5b-4347d62ce08b4f7fa32f48fb769bba68 received from {username='slave'} at 127.27.111.197:42227
I20240502 02:10:00.104688 7421 tablet_copy_service.cc:433] P ca5956dd8d174df99a28e38e8a7a2524: ending tablet copy session 85e53716ea13433289fe6435b6693a5b-4347d62ce08b4f7fa32f48fb769bba68 on tablet 4347d62ce08b4f7fa32f48fb769bba68 with peer 85e53716ea13433289fe6435b6693a5b
I20240502 02:10:00.104818 8633 ts_tablet_manager.cc:902] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da: Initiating tablet copy from peer ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893)
I20240502 02:10:00.105553 8633 tablet_copy_client.cc:308] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:40893
I20240502 02:10:00.109593 7421 tablet_copy_service.cc:140] P ca5956dd8d174df99a28e38e8a7a2524: Received BeginTabletCopySession request for tablet a7d090773bb34c6eb370f39ffae4e4ea from peer efabf88707da4c13b7ec85bbfd8a89da ({username='slave'} at 127.27.111.196:37125)
I20240502 02:10:00.109695 7421 tablet_copy_service.cc:161] P ca5956dd8d174df99a28e38e8a7a2524: Beginning new tablet copy session on tablet a7d090773bb34c6eb370f39ffae4e4ea from peer efabf88707da4c13b7ec85bbfd8a89da at {username='slave'} at 127.27.111.196:37125: session id = efabf88707da4c13b7ec85bbfd8a89da-a7d090773bb34c6eb370f39ffae4e4ea
I20240502 02:10:00.110452 7421 tablet_copy_source_session.cc:205] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:10:00.113354 8633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d090773bb34c6eb370f39ffae4e4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:00.118911 8633 tablet_copy_client.cc:783] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Starting download of 0 data blocks...
I20240502 02:10:00.119135 8633 tablet_copy_client.cc:647] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Starting download of 1 WAL segments...
I20240502 02:10:00.120537 8633 tablet_copy_client.cc:523] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:10:00.122010 8633 tablet_bootstrap.cc:492] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da: Bootstrap starting.
I20240502 02:10:00.127269 8633 log.cc:824] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da: Log is configured to *not* fsync() on all Append() calls
I20240502 02:10:00.135923 8625 ts_tablet_manager.cc:902] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b: Initiating tablet copy from peer defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:10:00.136435 8625 tablet_copy_client.cc:308] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:40407
I20240502 02:10:00.139946 7554 tablet_copy_service.cc:140] P defc209b6ddf4b719d89059a990ecb05: Received BeginTabletCopySession request for tablet 2ec2a56deaf64463aa75024fb5fa32f4 from peer 85e53716ea13433289fe6435b6693a5b ({username='slave'} at 127.27.111.197:43505)
I20240502 02:10:00.140087 7554 tablet_copy_service.cc:161] P defc209b6ddf4b719d89059a990ecb05: Beginning new tablet copy session on tablet 2ec2a56deaf64463aa75024fb5fa32f4 from peer 85e53716ea13433289fe6435b6693a5b at {username='slave'} at 127.27.111.197:43505: session id = 85e53716ea13433289fe6435b6693a5b-2ec2a56deaf64463aa75024fb5fa32f4
I20240502 02:10:00.141279 7554 tablet_copy_source_session.cc:205] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:10:00.142385 8625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec2a56deaf64463aa75024fb5fa32f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:00.144065 8625 tablet_copy_client.cc:783] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b: tablet copy: Starting download of 0 data blocks...
I20240502 02:10:00.144201 8625 tablet_copy_client.cc:647] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b: tablet copy: Starting download of 1 WAL segments...
I20240502 02:10:00.145435 8625 tablet_copy_client.cc:523] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:10:00.146354 8625 tablet_bootstrap.cc:492] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b: Bootstrap starting.
I20240502 02:10:00.148355 8633 tablet_bootstrap.cc:492] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:10:00.148649 8633 tablet_bootstrap.cc:492] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da: Bootstrap complete.
I20240502 02:10:00.148990 8633 ts_tablet_manager.cc:1367] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da: Time spent bootstrapping tablet: real 0.027s user 0.022s sys 0.004s
I20240502 02:10:00.151077 8633 raft_consensus.cc:348] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } }
I20240502 02:10:00.151319 8633 raft_consensus.cc:729] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efabf88707da4c13b7ec85bbfd8a89da, State: Initialized, Role: LEARNER
I20240502 02:10:00.151628 8633 consensus_queue.cc:260] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } }
I20240502 02:10:00.152473 8633 ts_tablet_manager.cc:1398] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 02:10:00.153187 7421 tablet_copy_service.cc:342] P ca5956dd8d174df99a28e38e8a7a2524: Request end of tablet copy session efabf88707da4c13b7ec85bbfd8a89da-a7d090773bb34c6eb370f39ffae4e4ea received from {username='slave'} at 127.27.111.196:37125
I20240502 02:10:00.153277 7421 tablet_copy_service.cc:433] P ca5956dd8d174df99a28e38e8a7a2524: ending tablet copy session efabf88707da4c13b7ec85bbfd8a89da-a7d090773bb34c6eb370f39ffae4e4ea on tablet a7d090773bb34c6eb370f39ffae4e4ea with peer efabf88707da4c13b7ec85bbfd8a89da
I20240502 02:10:00.161131 8633 ts_tablet_manager.cc:902] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da: Initiating tablet copy from peer defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:10:00.161679 8633 tablet_copy_client.cc:308] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:40407
I20240502 02:10:00.165155 7554 tablet_copy_service.cc:140] P defc209b6ddf4b719d89059a990ecb05: Received BeginTabletCopySession request for tablet c7c003a62d744c138c6ef685d826b7aa from peer efabf88707da4c13b7ec85bbfd8a89da ({username='slave'} at 127.27.111.196:33353)
I20240502 02:10:00.165266 7554 tablet_copy_service.cc:161] P defc209b6ddf4b719d89059a990ecb05: Beginning new tablet copy session on tablet c7c003a62d744c138c6ef685d826b7aa from peer efabf88707da4c13b7ec85bbfd8a89da at {username='slave'} at 127.27.111.196:33353: session id = efabf88707da4c13b7ec85bbfd8a89da-c7c003a62d744c138c6ef685d826b7aa
I20240502 02:10:00.165583 8625 tablet_bootstrap.cc:492] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:10:00.165859 8625 tablet_bootstrap.cc:492] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b: Bootstrap complete.
I20240502 02:10:00.166031 8625 ts_tablet_manager.cc:1367] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.004s
I20240502 02:10:00.166304 7554 tablet_copy_source_session.cc:205] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:10:00.166651 8625 raft_consensus.cc:348] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
I20240502 02:10:00.166812 8625 raft_consensus.cc:729] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85e53716ea13433289fe6435b6693a5b, State: Initialized, Role: LEARNER
I20240502 02:10:00.166923 8625 consensus_queue.cc:260] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
I20240502 02:10:00.167172 8633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c003a62d744c138c6ef685d826b7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:00.167219 8625 ts_tablet_manager.cc:1398] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 02:10:00.167769 7554 tablet_copy_service.cc:342] P defc209b6ddf4b719d89059a990ecb05: Request end of tablet copy session 85e53716ea13433289fe6435b6693a5b-2ec2a56deaf64463aa75024fb5fa32f4 received from {username='slave'} at 127.27.111.197:43505
I20240502 02:10:00.167876 7554 tablet_copy_service.cc:433] P defc209b6ddf4b719d89059a990ecb05: ending tablet copy session 85e53716ea13433289fe6435b6693a5b-2ec2a56deaf64463aa75024fb5fa32f4 on tablet 2ec2a56deaf64463aa75024fb5fa32f4 with peer 85e53716ea13433289fe6435b6693a5b
I20240502 02:10:00.169457 8633 tablet_copy_client.cc:783] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Starting download of 0 data blocks...
I20240502 02:10:00.169605 8633 tablet_copy_client.cc:647] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Starting download of 1 WAL segments...
I20240502 02:10:00.170608 8633 tablet_copy_client.cc:523] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:10:00.171531 8633 tablet_bootstrap.cc:492] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da: Bootstrap starting.
I20240502 02:10:00.173871 8625 ts_tablet_manager.cc:902] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b: Initiating tablet copy from peer 100889921252449596fc397b10fe903d (127.27.111.193:43115)
I20240502 02:10:00.174288 8625 tablet_copy_client.cc:308] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.193:43115
I20240502 02:10:00.177980 7289 tablet_copy_service.cc:140] P 100889921252449596fc397b10fe903d: Received BeginTabletCopySession request for tablet 7287522a25cc459099d78147ca1ed1cd from peer 85e53716ea13433289fe6435b6693a5b ({username='slave'} at 127.27.111.197:45365)
I20240502 02:10:00.178135 7289 tablet_copy_service.cc:161] P 100889921252449596fc397b10fe903d: Beginning new tablet copy session on tablet 7287522a25cc459099d78147ca1ed1cd from peer 85e53716ea13433289fe6435b6693a5b at {username='slave'} at 127.27.111.197:45365: session id = 85e53716ea13433289fe6435b6693a5b-7287522a25cc459099d78147ca1ed1cd
I20240502 02:10:00.179585 7289 tablet_copy_source_session.cc:205] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:10:00.180709 8625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7287522a25cc459099d78147ca1ed1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:00.182291 8625 tablet_copy_client.cc:783] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b: tablet copy: Starting download of 0 data blocks...
I20240502 02:10:00.182482 8625 tablet_copy_client.cc:647] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b: tablet copy: Starting download of 1 WAL segments...
I20240502 02:10:00.183859 8625 tablet_copy_client.cc:523] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:10:00.184780 8625 tablet_bootstrap.cc:492] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b: Bootstrap starting.
I20240502 02:10:00.189970 8633 tablet_bootstrap.cc:492] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:10:00.190222 8633 tablet_bootstrap.cc:492] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da: Bootstrap complete.
I20240502 02:10:00.190379 8633 ts_tablet_manager.cc:1367] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.000s
I20240502 02:10:00.190899 8633 raft_consensus.cc:348] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } }
I20240502 02:10:00.191005 8633 raft_consensus.cc:729] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efabf88707da4c13b7ec85bbfd8a89da, State: Initialized, Role: LEARNER
I20240502 02:10:00.191115 8633 consensus_queue.cc:260] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } }
I20240502 02:10:00.191342 8633 ts_tablet_manager.cc:1398] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:10:00.191926 7554 tablet_copy_service.cc:342] P defc209b6ddf4b719d89059a990ecb05: Request end of tablet copy session efabf88707da4c13b7ec85bbfd8a89da-c7c003a62d744c138c6ef685d826b7aa received from {username='slave'} at 127.27.111.196:33353
I20240502 02:10:00.192023 7554 tablet_copy_service.cc:433] P defc209b6ddf4b719d89059a990ecb05: ending tablet copy session efabf88707da4c13b7ec85bbfd8a89da-c7c003a62d744c138c6ef685d826b7aa on tablet c7c003a62d744c138c6ef685d826b7aa with peer efabf88707da4c13b7ec85bbfd8a89da
I20240502 02:10:00.203001 8625 tablet_bootstrap.cc:492] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:10:00.203331 8625 tablet_bootstrap.cc:492] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b: Bootstrap complete.
I20240502 02:10:00.203545 8625 ts_tablet_manager.cc:1367] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.001s
I20240502 02:10:00.204006 8625 raft_consensus.cc:348] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
I20240502 02:10:00.204109 8625 raft_consensus.cc:729] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85e53716ea13433289fe6435b6693a5b, State: Initialized, Role: LEARNER
I20240502 02:10:00.204193 8625 consensus_queue.cc:260] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
I20240502 02:10:00.204455 8625 ts_tablet_manager.cc:1398] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:10:00.205072 7289 tablet_copy_service.cc:342] P 100889921252449596fc397b10fe903d: Request end of tablet copy session 85e53716ea13433289fe6435b6693a5b-7287522a25cc459099d78147ca1ed1cd received from {username='slave'} at 127.27.111.197:45365
I20240502 02:10:00.205178 7289 tablet_copy_service.cc:433] P 100889921252449596fc397b10fe903d: ending tablet copy session 85e53716ea13433289fe6435b6693a5b-7287522a25cc459099d78147ca1ed1cd on tablet 7287522a25cc459099d78147ca1ed1cd with peer 85e53716ea13433289fe6435b6693a5b
I20240502 02:10:00.266800 8633 ts_tablet_manager.cc:902] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da: Initiating tablet copy from peer defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:10:00.267270 8633 tablet_copy_client.cc:308] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:40407
I20240502 02:10:00.267803 7554 tablet_copy_service.cc:140] P defc209b6ddf4b719d89059a990ecb05: Received BeginTabletCopySession request for tablet e6b4b64d18f34c0298369e247e670fcd from peer efabf88707da4c13b7ec85bbfd8a89da ({username='slave'} at 127.27.111.196:33353)
I20240502 02:10:00.267916 7554 tablet_copy_service.cc:161] P defc209b6ddf4b719d89059a990ecb05: Beginning new tablet copy session on tablet e6b4b64d18f34c0298369e247e670fcd from peer efabf88707da4c13b7ec85bbfd8a89da at {username='slave'} at 127.27.111.196:33353: session id = efabf88707da4c13b7ec85bbfd8a89da-e6b4b64d18f34c0298369e247e670fcd
I20240502 02:10:00.268674 7554 tablet_copy_source_session.cc:205] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:10:00.269483 8633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b4b64d18f34c0298369e247e670fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:00.271248 8633 tablet_copy_client.cc:783] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Starting download of 0 data blocks...
I20240502 02:10:00.271386 8633 tablet_copy_client.cc:647] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Starting download of 1 WAL segments...
I20240502 02:10:00.272500 8633 tablet_copy_client.cc:523] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:10:00.273628 8633 tablet_bootstrap.cc:492] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da: Bootstrap starting.
I20240502 02:10:00.291178 8633 tablet_bootstrap.cc:492] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:10:00.291482 8633 tablet_bootstrap.cc:492] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da: Bootstrap complete.
I20240502 02:10:00.291661 8633 ts_tablet_manager.cc:1367] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.004s
I20240502 02:10:00.292129 8633 raft_consensus.cc:348] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } }
I20240502 02:10:00.292239 8633 raft_consensus.cc:729] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efabf88707da4c13b7ec85bbfd8a89da, State: Initialized, Role: LEARNER
I20240502 02:10:00.292344 8633 consensus_queue.cc:260] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } }
I20240502 02:10:00.292632 8633 ts_tablet_manager.cc:1398] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:10:00.293102 7554 tablet_copy_service.cc:342] P defc209b6ddf4b719d89059a990ecb05: Request end of tablet copy session efabf88707da4c13b7ec85bbfd8a89da-e6b4b64d18f34c0298369e247e670fcd received from {username='slave'} at 127.27.111.196:33353
I20240502 02:10:00.293180 7554 tablet_copy_service.cc:433] P defc209b6ddf4b719d89059a990ecb05: ending tablet copy session efabf88707da4c13b7ec85bbfd8a89da-e6b4b64d18f34c0298369e247e670fcd on tablet e6b4b64d18f34c0298369e247e670fcd with peer efabf88707da4c13b7ec85bbfd8a89da
I20240502 02:10:00.295329 8625 ts_tablet_manager.cc:902] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b: Initiating tablet copy from peer ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893)
I20240502 02:10:00.295847 8625 tablet_copy_client.cc:308] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:40893
I20240502 02:10:00.296303 7421 tablet_copy_service.cc:140] P ca5956dd8d174df99a28e38e8a7a2524: Received BeginTabletCopySession request for tablet e7b56f82e754429e8cc8fc6d2b78ce8e from peer 85e53716ea13433289fe6435b6693a5b ({username='slave'} at 127.27.111.197:42227)
I20240502 02:10:00.296401 7421 tablet_copy_service.cc:161] P ca5956dd8d174df99a28e38e8a7a2524: Beginning new tablet copy session on tablet e7b56f82e754429e8cc8fc6d2b78ce8e from peer 85e53716ea13433289fe6435b6693a5b at {username='slave'} at 127.27.111.197:42227: session id = 85e53716ea13433289fe6435b6693a5b-e7b56f82e754429e8cc8fc6d2b78ce8e
I20240502 02:10:00.297122 7421 tablet_copy_source_session.cc:205] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:10:00.297876 8625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b56f82e754429e8cc8fc6d2b78ce8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:00.299659 8625 tablet_copy_client.cc:783] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b: tablet copy: Starting download of 0 data blocks...
I20240502 02:10:00.299810 8625 tablet_copy_client.cc:647] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b: tablet copy: Starting download of 1 WAL segments...
I20240502 02:10:00.300976 8625 tablet_copy_client.cc:523] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:10:00.301911 8625 tablet_bootstrap.cc:492] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b: Bootstrap starting.
I20240502 02:10:00.317530 8625 tablet_bootstrap.cc:492] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:10:00.317795 8625 tablet_bootstrap.cc:492] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b: Bootstrap complete.
I20240502 02:10:00.317951 8625 ts_tablet_manager.cc:1367] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20240502 02:10:00.318329 8625 raft_consensus.cc:348] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
I20240502 02:10:00.318481 8625 raft_consensus.cc:729] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85e53716ea13433289fe6435b6693a5b, State: Initialized, Role: LEARNER
I20240502 02:10:00.318605 8625 consensus_queue.cc:260] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
I20240502 02:10:00.318898 8625 ts_tablet_manager.cc:1398] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 02:10:00.319522 7421 tablet_copy_service.cc:342] P ca5956dd8d174df99a28e38e8a7a2524: Request end of tablet copy session 85e53716ea13433289fe6435b6693a5b-e7b56f82e754429e8cc8fc6d2b78ce8e received from {username='slave'} at 127.27.111.197:42227
I20240502 02:10:00.319609 7421 tablet_copy_service.cc:433] P ca5956dd8d174df99a28e38e8a7a2524: ending tablet copy session 85e53716ea13433289fe6435b6693a5b-e7b56f82e754429e8cc8fc6d2b78ce8e on tablet e7b56f82e754429e8cc8fc6d2b78ce8e with peer 85e53716ea13433289fe6435b6693a5b
I20240502 02:10:00.504459 8299 raft_consensus.cc:1203] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 02:10:00.542534 8299 raft_consensus.cc:1203] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 02:10:00.570714 8168 raft_consensus.cc:1203] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 02:10:00.683884 8168 raft_consensus.cc:1203] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 02:10:00.725596 8299 raft_consensus.cc:1203] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 02:10:00.741856 8168 raft_consensus.cc:1203] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 02:10:00.842185 8168 raft_consensus.cc:1203] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 02:10:00.988379 8740 raft_consensus.cc:1050] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05: attempting to promote NON_VOTER efabf88707da4c13b7ec85bbfd8a89da to VOTER
I20240502 02:10:00.988864 8740 consensus_queue.cc:237] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } }
I20240502 02:10:00.990236 7266 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:00.990242 7401 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:00.990322 8299 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:00.990758 8814 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:00.990972 8740 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:00.991158 8497 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:00.993341 8497 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:00.994616 8299 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:00.995294 7266 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:00.995316 7401 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:00.995983 7116 catalog_manager.cc:5484] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 reported cstate change: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:10:01.000363 7533 consensus_queue.cc:237] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } }
I20240502 02:10:01.001996 7266 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:01.002414 8497 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:01.002599 8299 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:01.002919 8497 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:01.003927 8692 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.004386 7266 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.005218 7109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c7c003a62d744c138c6ef685d826b7aa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:10:01.006016 8299 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.006232 7120 catalog_manager.cc:5484] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 reported cstate change: config changed from index 5 to 6, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:10:01.008945 7381 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c003a62d744c138c6ef685d826b7aa with delete_type TABLET_DATA_TOMBSTONED (TS ca5956dd8d174df99a28e38e8a7a2524 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44538
I20240502 02:10:01.009474 8822 tablet_replica.cc:330] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: stopping tablet replica
I20240502 02:10:01.009826 8822 raft_consensus.cc:2229] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:10:01.010138 8822 raft_consensus.cc:2258] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:10:01.011256 8822 ts_tablet_manager.cc:1875] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:10:01.013113 8822 ts_tablet_manager.cc:1888] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:10:01.013192 8822 log.cc:1196] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/c7c003a62d744c138c6ef685d826b7aa
I20240502 02:10:01.013739 7108 catalog_manager.cc:4830] TS ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893): tablet c7c003a62d744c138c6ef685d826b7aa (table rebalance_test_table_1 [id=789c28d35c724bb8a67e0674bacf590b]) successfully deleted
I20240502 02:10:01.022910 8739 raft_consensus.cc:1050] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524: attempting to promote NON_VOTER efabf88707da4c13b7ec85bbfd8a89da to VOTER
I20240502 02:10:01.023339 8739 consensus_queue.cc:237] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } }
I20240502 02:10:01.024878 7533 raft_consensus.cc:1261] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.025233 7266 raft_consensus.cc:1261] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.025346 8299 raft_consensus.cc:1261] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.025326 8738 consensus_queue.cc:1035] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.025763 8738 consensus_queue.cc:1035] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.025919 8739 consensus_queue.cc:1035] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.027822 8739 raft_consensus.cc:2932] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.028299 8299 raft_consensus.cc:2932] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.028493 7266 raft_consensus.cc:2932] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.030716 7533 raft_consensus.cc:2932] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.030938 7120 catalog_manager.cc:5484] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da reported cstate change: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.034099 7401 consensus_queue.cc:237] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } }
I20240502 02:10:01.035163 7266 raft_consensus.cc:1261] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:01.035225 8299 raft_consensus.cc:1261] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:01.035502 8738 consensus_queue.cc:1035] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:01.035663 8518 consensus_queue.cc:1035] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:01.037055 8739 raft_consensus.cc:2932] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER defc209b6ddf4b719d89059a990ecb05 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.037443 8299 raft_consensus.cc:2932] T a7d090773bb34c6eb370f39ffae4e4ea P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER defc209b6ddf4b719d89059a990ecb05 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.038460 7108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7d090773bb34c6eb370f39ffae4e4ea with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:10:01.038889 7266 raft_consensus.cc:2932] T a7d090773bb34c6eb370f39ffae4e4ea P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER defc209b6ddf4b719d89059a990ecb05 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.039198 7116 catalog_manager.cc:5484] T a7d090773bb34c6eb370f39ffae4e4ea P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: config changed from index 5 to 6, VOTER defc209b6ddf4b719d89059a990ecb05 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:10:01.041760 7511 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d090773bb34c6eb370f39ffae4e4ea with delete_type TABLET_DATA_TOMBSTONED (TS defc209b6ddf4b719d89059a990ecb05 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37144
I20240502 02:10:01.042291 8827 tablet_replica.cc:330] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05: stopping tablet replica
I20240502 02:10:01.042667 8827 raft_consensus.cc:2229] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:10:01.042902 8827 raft_consensus.cc:2258] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:10:01.044009 8827 ts_tablet_manager.cc:1875] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:10:01.046208 8827 ts_tablet_manager.cc:1888] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:10:01.046299 8827 log.cc:1196] T a7d090773bb34c6eb370f39ffae4e4ea P defc209b6ddf4b719d89059a990ecb05: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/a7d090773bb34c6eb370f39ffae4e4ea
I20240502 02:10:01.046885 7109 catalog_manager.cc:4830] TS defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407): tablet a7d090773bb34c6eb370f39ffae4e4ea (table rebalance_test_table_0 [id=424baa04ef41432ebdec6d07ed41cd5e]) successfully deleted
I20240502 02:10:01.086616 8739 raft_consensus.cc:1050] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524: attempting to promote NON_VOTER 85e53716ea13433289fe6435b6693a5b to VOTER
I20240502 02:10:01.086993 8739 consensus_queue.cc:237] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } }
I20240502 02:10:01.088348 8168 raft_consensus.cc:1261] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.088366 7533 raft_consensus.cc:1261] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.088490 7266 raft_consensus.cc:1261] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.088852 8518 consensus_queue.cc:1035] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.089053 8739 consensus_queue.cc:1035] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.089212 8829 consensus_queue.cc:1035] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.091177 8739 raft_consensus.cc:2932] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.091576 7533 raft_consensus.cc:2932] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.092394 8168 raft_consensus.cc:2932] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.093176 7266 raft_consensus.cc:2932] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.094300 7124 catalog_manager.cc:5484] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 reported cstate change: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.097572 7401 consensus_queue.cc:237] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } }
I20240502 02:10:01.098685 8168 raft_consensus.cc:1261] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:01.098738 7266 raft_consensus.cc:1261] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:01.099056 8518 consensus_queue.cc:1035] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:01.099354 8518 consensus_queue.cc:1035] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:01.100804 8738 raft_consensus.cc:2932] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER defc209b6ddf4b719d89059a990ecb05 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.101207 8168 raft_consensus.cc:2932] T 4347d62ce08b4f7fa32f48fb769bba68 P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER defc209b6ddf4b719d89059a990ecb05 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.101199 7266 raft_consensus.cc:2932] T 4347d62ce08b4f7fa32f48fb769bba68 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER defc209b6ddf4b719d89059a990ecb05 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.102314 7108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4347d62ce08b4f7fa32f48fb769bba68 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:10:01.103147 7120 catalog_manager.cc:5484] T 4347d62ce08b4f7fa32f48fb769bba68 P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: config changed from index 5 to 6, VOTER defc209b6ddf4b719d89059a990ecb05 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:10:01.105612 7511 tablet_service.cc:1514] Processing DeleteTablet for tablet 4347d62ce08b4f7fa32f48fb769bba68 with delete_type TABLET_DATA_TOMBSTONED (TS defc209b6ddf4b719d89059a990ecb05 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37144
I20240502 02:10:01.105921 8814 raft_consensus.cc:1050] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05: attempting to promote NON_VOTER efabf88707da4c13b7ec85bbfd8a89da to VOTER
I20240502 02:10:01.106335 8814 consensus_queue.cc:237] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } }
I20240502 02:10:01.106637 8827 tablet_replica.cc:330] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05: stopping tablet replica
I20240502 02:10:01.106772 8827 raft_consensus.cc:2229] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:10:01.106938 8827 raft_consensus.cc:2258] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:10:01.107802 8299 raft_consensus.cc:1261] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.107816 7401 raft_consensus.cc:1261] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.108145 7266 raft_consensus.cc:1261] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.108186 8692 consensus_queue.cc:1035] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.108355 8740 consensus_queue.cc:1035] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.108606 8740 consensus_queue.cc:1035] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.108812 8827 ts_tablet_manager.cc:1875] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:10:01.110446 8740 raft_consensus.cc:2932] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.110972 7266 raft_consensus.cc:2932] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.111549 8827 ts_tablet_manager.cc:1888] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:10:01.111624 8827 log.cc:1196] T 4347d62ce08b4f7fa32f48fb769bba68 P defc209b6ddf4b719d89059a990ecb05: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/4347d62ce08b4f7fa32f48fb769bba68
I20240502 02:10:01.112241 7401 raft_consensus.cc:2932] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.112874 7109 catalog_manager.cc:4830] TS defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407): tablet 4347d62ce08b4f7fa32f48fb769bba68 (table rebalance_test_table_2 [id=178c22953cac45a28113df2575de7ad7]) successfully deleted
I20240502 02:10:01.113485 7122 catalog_manager.cc:5484] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 reported cstate change: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 02:10:01.114408 8299 raft_consensus.cc:2932] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.117849 7533 consensus_queue.cc:237] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } }
I20240502 02:10:01.119011 7401 raft_consensus.cc:1261] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:01.119035 8299 raft_consensus.cc:1261] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:01.119426 8740 consensus_queue.cc:1035] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:01.119599 8692 consensus_queue.cc:1035] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:01.121068 8692 raft_consensus.cc:2932] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.121539 8299 raft_consensus.cc:2932] T e6b4b64d18f34c0298369e247e670fcd P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.121549 7401 raft_consensus.cc:2932] T e6b4b64d18f34c0298369e247e670fcd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:01.122752 7109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6b4b64d18f34c0298369e247e670fcd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:10:01.123942 7116 catalog_manager.cc:5484] T e6b4b64d18f34c0298369e247e670fcd P defc209b6ddf4b719d89059a990ecb05 reported cstate change: config changed from index 5 to 6, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:10:01.126332 7247 tablet_service.cc:1514] Processing DeleteTablet for tablet e6b4b64d18f34c0298369e247e670fcd with delete_type TABLET_DATA_TOMBSTONED (TS 100889921252449596fc397b10fe903d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44892
I20240502 02:10:01.126941 8837 tablet_replica.cc:330] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d: stopping tablet replica
I20240502 02:10:01.127300 8837 raft_consensus.cc:2229] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:10:01.127596 8837 raft_consensus.cc:2258] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:10:01.128721 8837 ts_tablet_manager.cc:1875] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:10:01.130630 8837 ts_tablet_manager.cc:1888] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:10:01.130710 8837 log.cc:1196] T e6b4b64d18f34c0298369e247e670fcd P 100889921252449596fc397b10fe903d: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/e6b4b64d18f34c0298369e247e670fcd
I20240502 02:10:01.131273 7110 catalog_manager.cc:4830] TS 100889921252449596fc397b10fe903d (127.27.111.193:43115): tablet e6b4b64d18f34c0298369e247e670fcd (table rebalance_test_table_2 [id=178c22953cac45a28113df2575de7ad7]) successfully deleted
I20240502 02:10:01.254546 8888 raft_consensus.cc:1050] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d: attempting to promote NON_VOTER 85e53716ea13433289fe6435b6693a5b to VOTER
I20240502 02:10:01.255069 8888 consensus_queue.cc:237] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } }
I20240502 02:10:01.256511 8168 raft_consensus.cc:1261] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Refusing update from remote peer 100889921252449596fc397b10fe903d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.256614 7533 raft_consensus.cc:1261] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer 100889921252449596fc397b10fe903d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.256685 7401 raft_consensus.cc:1261] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer 100889921252449596fc397b10fe903d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.257015 8890 consensus_queue.cc:1035] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.257223 8752 consensus_queue.cc:1035] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.257412 8891 consensus_queue.cc:1035] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.259544 8752 raft_consensus.cc:2932] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.260023 7401 raft_consensus.cc:2932] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.260396 7533 raft_consensus.cc:2932] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.261832 8168 raft_consensus.cc:2932] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.262892 7122 catalog_manager.cc:5484] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "100889921252449596fc397b10fe903d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.266109 7266 consensus_queue.cc:237] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } }
I20240502 02:10:01.267397 7533 raft_consensus.cc:1261] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer 100889921252449596fc397b10fe903d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:01.267580 8168 raft_consensus.cc:1261] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Refusing update from remote peer 100889921252449596fc397b10fe903d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:01.267793 8891 consensus_queue.cc:1035] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:01.267956 8752 consensus_queue.cc:1035] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:01.269409 8890 raft_consensus.cc:2932] T 7287522a25cc459099d78147ca1ed1cd P 100889921252449596fc397b10fe903d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.269793 7533 raft_consensus.cc:2932] T 7287522a25cc459099d78147ca1ed1cd P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.269932 8168 raft_consensus.cc:2932] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.270741 7110 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7287522a25cc459099d78147ca1ed1cd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:10:01.272126 7120 catalog_manager.cc:5484] T 7287522a25cc459099d78147ca1ed1cd P 85e53716ea13433289fe6435b6693a5b reported cstate change: config changed from index 5 to 6, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "100889921252449596fc397b10fe903d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.274226 7381 tablet_service.cc:1514] Processing DeleteTablet for tablet 7287522a25cc459099d78147ca1ed1cd with delete_type TABLET_DATA_TOMBSTONED (TS ca5956dd8d174df99a28e38e8a7a2524 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44538
I20240502 02:10:01.274555 8822 tablet_replica.cc:330] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524: stopping tablet replica
I20240502 02:10:01.275038 8822 raft_consensus.cc:2229] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:10:01.275183 8822 raft_consensus.cc:2258] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:10:01.275772 8822 ts_tablet_manager.cc:1875] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:10:01.277510 8822 ts_tablet_manager.cc:1888] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:10:01.277581 8822 log.cc:1196] T 7287522a25cc459099d78147ca1ed1cd P ca5956dd8d174df99a28e38e8a7a2524: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/7287522a25cc459099d78147ca1ed1cd
I20240502 02:10:01.278090 7108 catalog_manager.cc:4830] TS ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893): tablet 7287522a25cc459099d78147ca1ed1cd (table rebalance_test_table_1 [id=789c28d35c724bb8a67e0674bacf590b]) successfully deleted
I20240502 02:10:01.360153 8497 raft_consensus.cc:1050] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05: attempting to promote NON_VOTER 85e53716ea13433289fe6435b6693a5b to VOTER
I20240502 02:10:01.360563 8497 consensus_queue.cc:237] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } }
I20240502 02:10:01.362027 7401 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.362004 8168 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.362030 7266 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.362447 8814 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.362627 8740 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.362818 8692 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.364790 8814 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.365242 7266 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.365422 7401 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.366814 8168 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.368393 7116 catalog_manager.cc:5484] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.371608 7533 consensus_queue.cc:237] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } }
I20240502 02:10:01.372689 8168 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:01.372977 7401 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:01.373085 8814 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:01.373371 8740 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:01.374891 8692 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.375237 8168 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.376425 7109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ec2a56deaf64463aa75024fb5fa32f4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:10:01.376706 7401 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.377257 7122 catalog_manager.cc:5484] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 reported cstate change: config changed from index 5 to 6, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:10:01.377434 8518 raft_consensus.cc:1050] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524: attempting to promote NON_VOTER 85e53716ea13433289fe6435b6693a5b to VOTER
I20240502 02:10:01.377736 8518 consensus_queue.cc:237] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } }
I20240502 02:10:01.379566 8168 raft_consensus.cc:1261] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.380000 8829 consensus_queue.cc:1035] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.380249 7266 raft_consensus.cc:1261] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.380402 7247 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ec2a56deaf64463aa75024fb5fa32f4 with delete_type TABLET_DATA_TOMBSTONED (TS 100889921252449596fc397b10fe903d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44892
I20240502 02:10:01.380615 8837 tablet_replica.cc:330] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: stopping tablet replica
I20240502 02:10:01.380681 8829 consensus_queue.cc:1035] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.380754 8837 raft_consensus.cc:2229] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:10:01.380916 8837 raft_consensus.cc:2258] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:10:01.381989 7533 raft_consensus.cc:1261] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:01.382911 8518 consensus_queue.cc:1035] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:01.382869 8837 ts_tablet_manager.cc:1875] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:10:01.383352 8900 raft_consensus.cc:2932] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.383813 8168 raft_consensus.cc:2932] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.385402 8837 ts_tablet_manager.cc:1888] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:10:01.385473 7268 raft_consensus.cc:2932] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.385886 7533 raft_consensus.cc:2932] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.386142 7120 catalog_manager.cc:5484] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:10:01.385494 8837 log.cc:1196] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/2ec2a56deaf64463aa75024fb5fa32f4
I20240502 02:10:01.388999 7110 catalog_manager.cc:4830] TS 100889921252449596fc397b10fe903d (127.27.111.193:43115): tablet 2ec2a56deaf64463aa75024fb5fa32f4 (table rebalance_test_table_0 [id=424baa04ef41432ebdec6d07ed41cd5e]) successfully deleted
I20240502 02:10:01.389707 7401 consensus_queue.cc:237] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } }
I20240502 02:10:01.390765 7533 raft_consensus.cc:1261] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:01.390767 8168 raft_consensus.cc:1261] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:01.391124 8518 consensus_queue.cc:1035] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:01.391408 8518 consensus_queue.cc:1035] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:01.392551 8518 raft_consensus.cc:2932] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.392984 7533 raft_consensus.cc:2932] T e7b56f82e754429e8cc8fc6d2b78ce8e P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.394039 7108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e7b56f82e754429e8cc8fc6d2b78ce8e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:10:01.394843 8168 raft_consensus.cc:2932] T e7b56f82e754429e8cc8fc6d2b78ce8e P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:01.395320 7124 catalog_manager.cc:5484] T e7b56f82e754429e8cc8fc6d2b78ce8e P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: config changed from index 5 to 6, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 02:10:01.397614 7247 tablet_service.cc:1514] Processing DeleteTablet for tablet e7b56f82e754429e8cc8fc6d2b78ce8e with delete_type TABLET_DATA_TOMBSTONED (TS 100889921252449596fc397b10fe903d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44892
I20240502 02:10:01.398000 8837 tablet_replica.cc:330] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d: stopping tablet replica
I20240502 02:10:01.398139 8837 raft_consensus.cc:2229] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:10:01.398277 8837 raft_consensus.cc:2258] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:10:01.399101 8837 ts_tablet_manager.cc:1875] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:10:01.400882 8837 ts_tablet_manager.cc:1888] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:10:01.400960 8837 log.cc:1196] T e7b56f82e754429e8cc8fc6d2b78ce8e P 100889921252449596fc397b10fe903d: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/e7b56f82e754429e8cc8fc6d2b78ce8e
I20240502 02:10:01.401458 7110 catalog_manager.cc:4830] TS 100889921252449596fc397b10fe903d (127.27.111.193:43115): tablet e7b56f82e754429e8cc8fc6d2b78ce8e (table rebalance_test_table_0 [id=424baa04ef41432ebdec6d07ed41cd5e]) successfully deleted
I20240502 02:10:01.478556 7381 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:01.480150 8279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:10:01.480396 7247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:10:01.480891 8148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:10:01.486718 7511 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:01.521183 7533 consensus_queue.cc:237] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } }
I20240502 02:10:01.522733 7268 raft_consensus.cc:1261] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:10:01.522744 7401 raft_consensus.cc:1261] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 02:10:01.522974 7467 consensus_peers.cc:485] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 -> Peer efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196:45431): Couldn't send request to peer efabf88707da4c13b7ec85bbfd8a89da. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00e19846903d4a088e91ba3efaaa6305. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:10:01.523142 8814 consensus_queue.cc:1035] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:10:01.523569 8814 consensus_queue.cc:1035] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:10:01.524928 8740 raft_consensus.cc:2932] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } } }
I20240502 02:10:01.525365 7401 raft_consensus.cc:2932] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } } }
I20240502 02:10:01.525826 7268 raft_consensus.cc:2932] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } } }
I20240502 02:10:01.528342 7124 catalog_manager.cc:5484] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: config changed from index -1 to 4, NON_VOTER efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) added. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } } }
I20240502 02:10:01.535591 7401 consensus_queue.cc:237] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
I20240502 02:10:01.536888 7268 raft_consensus.cc:1261] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:10:01.537232 7533 raft_consensus.cc:1261] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:10:01.537324 8900 consensus_queue.cc:1035] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:10:01.537626 8829 consensus_queue.cc:1035] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 02:10:01.537806 7337 consensus_peers.cc:485] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 -> Peer 85e53716ea13433289fe6435b6693a5b (127.27.111.197:36289): Couldn't send request to peer 85e53716ea13433289fe6435b6693a5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a048c9ba70c40e38c3dc38fd997c1bf. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:10:01.538985 8829 raft_consensus.cc:2932] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
I20240502 02:10:01.539498 7268 raft_consensus.cc:2932] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
I20240502 02:10:01.539702 7533 raft_consensus.cc:2932] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } } }
I20240502 02:10:01.541256 7122 catalog_manager.cc:5484] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: config changed from index -1 to 4, NON_VOTER 85e53716ea13433289fe6435b6693a5b (127.27.111.197) added. New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:10:02.106277 8996 ts_tablet_manager.cc:902] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b: Initiating tablet copy from peer ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893)
I20240502 02:10:02.106827 8996 tablet_copy_client.cc:308] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.194:40893
I20240502 02:10:02.107355 7421 tablet_copy_service.cc:140] P ca5956dd8d174df99a28e38e8a7a2524: Received BeginTabletCopySession request for tablet 4a048c9ba70c40e38c3dc38fd997c1bf from peer 85e53716ea13433289fe6435b6693a5b ({username='slave'} at 127.27.111.197:42227)
I20240502 02:10:02.107474 7421 tablet_copy_service.cc:161] P ca5956dd8d174df99a28e38e8a7a2524: Beginning new tablet copy session on tablet 4a048c9ba70c40e38c3dc38fd997c1bf from peer 85e53716ea13433289fe6435b6693a5b at {username='slave'} at 127.27.111.197:42227: session id = 85e53716ea13433289fe6435b6693a5b-4a048c9ba70c40e38c3dc38fd997c1bf
I20240502 02:10:02.108258 7421 tablet_copy_source_session.cc:205] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:10:02.109153 8996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a048c9ba70c40e38c3dc38fd997c1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:02.111081 8996 tablet_copy_client.cc:783] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b: tablet copy: Starting download of 0 data blocks...
I20240502 02:10:02.111244 8996 tablet_copy_client.cc:647] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b: tablet copy: Starting download of 1 WAL segments...
I20240502 02:10:02.112475 8996 tablet_copy_client.cc:523] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:10:02.113580 8996 tablet_bootstrap.cc:492] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b: Bootstrap starting.
I20240502 02:10:02.131413 8996 tablet_bootstrap.cc:492] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:10:02.131732 8996 tablet_bootstrap.cc:492] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b: Bootstrap complete.
I20240502 02:10:02.131915 8996 ts_tablet_manager.cc:1367] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.003s
I20240502 02:10:02.132337 8996 raft_consensus.cc:348] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
I20240502 02:10:02.132455 8996 raft_consensus.cc:729] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85e53716ea13433289fe6435b6693a5b, State: Initialized, Role: LEARNER
I20240502 02:10:02.132565 8996 consensus_queue.cc:260] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: NON_VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: true } }
I20240502 02:10:02.132903 8996 ts_tablet_manager.cc:1398] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 02:10:02.133513 7421 tablet_copy_service.cc:342] P ca5956dd8d174df99a28e38e8a7a2524: Request end of tablet copy session 85e53716ea13433289fe6435b6693a5b-4a048c9ba70c40e38c3dc38fd997c1bf received from {username='slave'} at 127.27.111.197:42227
I20240502 02:10:02.133617 7421 tablet_copy_service.cc:433] P ca5956dd8d174df99a28e38e8a7a2524: ending tablet copy session 85e53716ea13433289fe6435b6693a5b-4a048c9ba70c40e38c3dc38fd997c1bf on tablet 4a048c9ba70c40e38c3dc38fd997c1bf with peer 85e53716ea13433289fe6435b6693a5b
I20240502 02:10:02.147934 9003 ts_tablet_manager.cc:902] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da: Initiating tablet copy from peer defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:10:02.148443 9003 tablet_copy_client.cc:308] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:40407
I20240502 02:10:02.148980 7554 tablet_copy_service.cc:140] P defc209b6ddf4b719d89059a990ecb05: Received BeginTabletCopySession request for tablet 00e19846903d4a088e91ba3efaaa6305 from peer efabf88707da4c13b7ec85bbfd8a89da ({username='slave'} at 127.27.111.196:33353)
I20240502 02:10:02.149098 7554 tablet_copy_service.cc:161] P defc209b6ddf4b719d89059a990ecb05: Beginning new tablet copy session on tablet 00e19846903d4a088e91ba3efaaa6305 from peer efabf88707da4c13b7ec85bbfd8a89da at {username='slave'} at 127.27.111.196:33353: session id = efabf88707da4c13b7ec85bbfd8a89da-00e19846903d4a088e91ba3efaaa6305
I20240502 02:10:02.149878 7554 tablet_copy_source_session.cc:205] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:10:02.150774 9003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00e19846903d4a088e91ba3efaaa6305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:02.152681 9003 tablet_copy_client.cc:783] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Starting download of 0 data blocks...
I20240502 02:10:02.152822 9003 tablet_copy_client.cc:647] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Starting download of 1 WAL segments...
I20240502 02:10:02.153968 9003 tablet_copy_client.cc:523] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:10:02.155079 9003 tablet_bootstrap.cc:492] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da: Bootstrap starting.
I20240502 02:10:02.171972 9003 tablet_bootstrap.cc:492] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:10:02.172300 9003 tablet_bootstrap.cc:492] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da: Bootstrap complete.
I20240502 02:10:02.172482 9003 ts_tablet_manager.cc:1367] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.003s
I20240502 02:10:02.173187 9003 raft_consensus.cc:348] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } }
I20240502 02:10:02.173313 9003 raft_consensus.cc:729] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efabf88707da4c13b7ec85bbfd8a89da, State: Initialized, Role: LEARNER
I20240502 02:10:02.173435 9003 consensus_queue.cc:260] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: NON_VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: true } }
I20240502 02:10:02.173803 9003 ts_tablet_manager.cc:1398] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 02:10:02.174526 7554 tablet_copy_service.cc:342] P defc209b6ddf4b719d89059a990ecb05: Request end of tablet copy session efabf88707da4c13b7ec85bbfd8a89da-00e19846903d4a088e91ba3efaaa6305 received from {username='slave'} at 127.27.111.196:33353
I20240502 02:10:02.174633 7554 tablet_copy_service.cc:433] P defc209b6ddf4b719d89059a990ecb05: ending tablet copy session efabf88707da4c13b7ec85bbfd8a89da-00e19846903d4a088e91ba3efaaa6305 on tablet 00e19846903d4a088e91ba3efaaa6305 with peer efabf88707da4c13b7ec85bbfd8a89da
I20240502 02:10:02.629316 8168 raft_consensus.cc:1203] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 02:10:02.750593 8299 raft_consensus.cc:1203] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 02:10:03.090415 9041 raft_consensus.cc:1050] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524: attempting to promote NON_VOTER 85e53716ea13433289fe6435b6693a5b to VOTER
I20240502 02:10:03.090894 9041 consensus_queue.cc:237] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } }
I20240502 02:10:03.092345 8168 raft_consensus.cc:1261] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b [term 1 LEARNER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:03.092373 7268 raft_consensus.cc:1261] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:03.092373 7533 raft_consensus.cc:1261] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:03.092803 9040 consensus_queue.cc:1035] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:03.093031 9041 consensus_queue.cc:1035] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:03.093245 8900 consensus_queue.cc:1035] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:03.095289 9040 raft_consensus.cc:2932] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:03.095747 8168 raft_consensus.cc:2932] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:03.095788 7533 raft_consensus.cc:2932] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:03.097533 7268 raft_consensus.cc:2932] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:03.098140 7120 catalog_manager.cc:5484] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b reported cstate change: config changed from index 4 to 5, 85e53716ea13433289fe6435b6693a5b (127.27.111.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:03.103300 7401 consensus_queue.cc:237] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } }
I20240502 02:10:03.104524 8168 raft_consensus.cc:1261] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:03.104594 7268 raft_consensus.cc:1261] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer ca5956dd8d174df99a28e38e8a7a2524: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:03.104887 9041 consensus_queue.cc:1035] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:03.105060 9040 consensus_queue.cc:1035] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:03.106563 9040 raft_consensus.cc:2932] T 4a048c9ba70c40e38c3dc38fd997c1bf P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER defc209b6ddf4b719d89059a990ecb05 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:03.107123 8168 raft_consensus.cc:2932] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER defc209b6ddf4b719d89059a990ecb05 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:03.107371 7268 raft_consensus.cc:2932] T 4a048c9ba70c40e38c3dc38fd997c1bf P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER defc209b6ddf4b719d89059a990ecb05 (127.27.111.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:03.109319 7108 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4a048c9ba70c40e38c3dc38fd997c1bf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:10:03.109272 7116 catalog_manager.cc:5484] T 4a048c9ba70c40e38c3dc38fd997c1bf P 85e53716ea13433289fe6435b6693a5b reported cstate change: config changed from index 5 to 6, VOTER defc209b6ddf4b719d89059a990ecb05 (127.27.111.195) evicted. New cstate: current_term: 1 leader_uuid: "ca5956dd8d174df99a28e38e8a7a2524" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } }
I20240502 02:10:03.112000 7511 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a048c9ba70c40e38c3dc38fd997c1bf with delete_type TABLET_DATA_TOMBSTONED (TS defc209b6ddf4b719d89059a990ecb05 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37144
I20240502 02:10:03.112376 9075 tablet_replica.cc:330] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05: stopping tablet replica
I20240502 02:10:03.112672 9075 raft_consensus.cc:2229] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:10:03.112845 9075 raft_consensus.cc:2258] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:10:03.113474 9075 ts_tablet_manager.cc:1875] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:10:03.115381 9075 ts_tablet_manager.cc:1888] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:10:03.115454 9075 log.cc:1196] T 4a048c9ba70c40e38c3dc38fd997c1bf P defc209b6ddf4b719d89059a990ecb05: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/wals/4a048c9ba70c40e38c3dc38fd997c1bf
I20240502 02:10:03.115964 7109 catalog_manager.cc:4830] TS defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407): tablet 4a048c9ba70c40e38c3dc38fd997c1bf (table rebalance_test_table_1 [id=789c28d35c724bb8a67e0674bacf590b]) successfully deleted
I20240502 02:10:03.249179 9043 raft_consensus.cc:1050] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05: attempting to promote NON_VOTER efabf88707da4c13b7ec85bbfd8a89da to VOTER
I20240502 02:10:03.249590 9043 consensus_queue.cc:237] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } }
I20240502 02:10:03.251114 8299 raft_consensus.cc:1261] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da [term 1 LEARNER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:03.251350 7401 raft_consensus.cc:1261] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:03.251394 7268 raft_consensus.cc:1261] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:10:03.251787 9078 consensus_queue.cc:1035] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:03.252024 9077 consensus_queue.cc:1035] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:03.252216 9043 consensus_queue.cc:1035] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:10:03.254817 7401 raft_consensus.cc:2932] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:03.256194 7268 raft_consensus.cc:2932] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:03.254354 9077 raft_consensus.cc:2932] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:03.259071 7116 catalog_manager.cc:5484] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d reported cstate change: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:03.261051 8299 raft_consensus.cc:2932] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efabf88707da4c13b7ec85bbfd8a89da (127.27.111.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:03.265875 7533 consensus_queue.cc:237] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } }
I20240502 02:10:03.267166 8299 raft_consensus.cc:1261] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:03.267225 7268 raft_consensus.cc:1261] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:10:03.267588 8814 consensus_queue.cc:1035] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:03.267808 9077 consensus_queue.cc:1035] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:10:03.269533 9043 raft_consensus.cc:2932] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:03.270992 7109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00e19846903d4a088e91ba3efaaa6305 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:10:03.271381 8299 raft_consensus.cc:2932] T 00e19846903d4a088e91ba3efaaa6305 P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:03.271399 7268 raft_consensus.cc:2932] T 00e19846903d4a088e91ba3efaaa6305 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } }
I20240502 02:10:03.273326 7124 catalog_manager.cc:5484] T 00e19846903d4a088e91ba3efaaa6305 P defc209b6ddf4b719d89059a990ecb05 reported cstate change: config changed from index 5 to 6, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:10:03.276278 7381 tablet_service.cc:1514] Processing DeleteTablet for tablet 00e19846903d4a088e91ba3efaaa6305 with delete_type TABLET_DATA_TOMBSTONED (TS ca5956dd8d174df99a28e38e8a7a2524 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44538
I20240502 02:10:03.276736 9100 tablet_replica.cc:330] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524: stopping tablet replica
I20240502 02:10:03.276980 9100 raft_consensus.cc:2229] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:10:03.277266 9100 raft_consensus.cc:2258] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:10:03.278198 9100 ts_tablet_manager.cc:1875] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:10:03.280321 9100 ts_tablet_manager.cc:1888] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:10:03.280416 9100 log.cc:1196] T 00e19846903d4a088e91ba3efaaa6305 P ca5956dd8d174df99a28e38e8a7a2524: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/00e19846903d4a088e91ba3efaaa6305
I20240502 02:10:03.280974 7108 catalog_manager.cc:4830] TS ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893): tablet 00e19846903d4a088e91ba3efaaa6305 (table rebalance_test_table_2 [id=178c22953cac45a28113df2575de7ad7]) successfully deleted
I20240502 02:10:03.330498 7511 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:03.330528 8148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:10:03.330837 7381 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:03.332269 7247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:10:03.337831 8279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:10:03.427151 7247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:10:03.429430 8148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:10:03.430158 7381 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:03.431573 8279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:10:03.453310 7511 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:03.480784 7533 consensus_queue.cc:237] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: NON_VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: true } }
I20240502 02:10:03.482259 8168 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 02:10:03.482570 7468 consensus_peers.cc:485] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 -> Peer 100889921252449596fc397b10fe903d (127.27.111.193:43115): Couldn't send request to peer 100889921252449596fc397b10fe903d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:10:03.482646 7401 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:10:03.483073 9078 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:10:03.483279 8814 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:10:03.485050 9078 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 100889921252449596fc397b10fe903d (127.27.111.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: NON_VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: true } } }
I20240502 02:10:03.485718 8168 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 100889921252449596fc397b10fe903d (127.27.111.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: NON_VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: true } } }
I20240502 02:10:03.488266 7122 catalog_manager.cc:5484] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 reported cstate change: config changed from index 6 to 7, NON_VOTER 100889921252449596fc397b10fe903d (127.27.111.193) added. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: NON_VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:10:03.488639 7401 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 100889921252449596fc397b10fe903d (127.27.111.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: NON_VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: true } } }
I20240502 02:10:03.495626 7533 consensus_queue.cc:237] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: NON_VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: true } }
I20240502 02:10:03.497040 7268 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 02:10:03.497227 7466 consensus_peers.cc:485] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 -> Peer ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893): Couldn't send request to peer ca5956dd8d174df99a28e38e8a7a2524. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:10:03.497244 8299 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:10:03.497541 9043 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:10:03.497735 8814 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:10:03.499511 8814 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: NON_VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: true } } }
I20240502 02:10:03.499959 8299 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: NON_VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: true } } }
I20240502 02:10:03.500548 7268 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: NON_VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: true } } }
I20240502 02:10:03.502847 7124 catalog_manager.cc:5484] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da reported cstate change: config changed from index 6 to 7, NON_VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) added. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: NON_VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: true } } }
I20240502 02:10:03.943609 9213 ts_tablet_manager.cc:902] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Initiating tablet copy from peer defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:10:03.944312 9213 tablet_copy_client.cc:308] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:40407
I20240502 02:10:03.944859 7554 tablet_copy_service.cc:140] P defc209b6ddf4b719d89059a990ecb05: Received BeginTabletCopySession request for tablet 2ec2a56deaf64463aa75024fb5fa32f4 from peer 100889921252449596fc397b10fe903d ({username='slave'} at 127.27.111.193:40495)
I20240502 02:10:03.944962 7554 tablet_copy_service.cc:161] P defc209b6ddf4b719d89059a990ecb05: Beginning new tablet copy session on tablet 2ec2a56deaf64463aa75024fb5fa32f4 from peer 100889921252449596fc397b10fe903d at {username='slave'} at 127.27.111.193:40495: session id = 100889921252449596fc397b10fe903d-2ec2a56deaf64463aa75024fb5fa32f4
I20240502 02:10:03.945647 7554 tablet_copy_source_session.cc:205] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:10:03.946605 9213 ts_tablet_manager.cc:1875] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:10:03.948575 9213 ts_tablet_manager.cc:1888] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 02:10:03.948737 9213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec2a56deaf64463aa75024fb5fa32f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:03.949867 9213 tablet_copy_client.cc:783] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: tablet copy: Starting download of 0 data blocks...
I20240502 02:10:03.950057 9213 tablet_copy_client.cc:647] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: tablet copy: Starting download of 1 WAL segments...
I20240502 02:10:03.951537 9213 tablet_copy_client.cc:523] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:10:03.952886 9213 tablet_bootstrap.cc:492] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Bootstrap starting.
I20240502 02:10:03.972167 9213 tablet_bootstrap.cc:492] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:10:03.972551 9213 tablet_bootstrap.cc:492] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Bootstrap complete.
I20240502 02:10:03.972730 9213 ts_tablet_manager.cc:1367] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20240502 02:10:03.973157 9213 raft_consensus.cc:348] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: NON_VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: true } }
I20240502 02:10:03.973271 9213 raft_consensus.cc:729] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 100889921252449596fc397b10fe903d, State: Initialized, Role: LEARNER
I20240502 02:10:03.973388 9213 consensus_queue.cc:260] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: NON_VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: true } }
I20240502 02:10:03.973657 9213 ts_tablet_manager.cc:1398] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:10:03.974375 7554 tablet_copy_service.cc:342] P defc209b6ddf4b719d89059a990ecb05: Request end of tablet copy session 100889921252449596fc397b10fe903d-2ec2a56deaf64463aa75024fb5fa32f4 received from {username='slave'} at 127.27.111.193:40495
I20240502 02:10:03.974501 7554 tablet_copy_service.cc:433] P defc209b6ddf4b719d89059a990ecb05: ending tablet copy session 100889921252449596fc397b10fe903d-2ec2a56deaf64463aa75024fb5fa32f4 on tablet 2ec2a56deaf64463aa75024fb5fa32f4 with peer 100889921252449596fc397b10fe903d
I20240502 02:10:03.991466 9219 ts_tablet_manager.cc:902] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: Initiating tablet copy from peer defc209b6ddf4b719d89059a990ecb05 (127.27.111.195:40407)
I20240502 02:10:03.992161 9219 tablet_copy_client.cc:308] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: tablet copy: Beginning tablet copy session from remote peer at address 127.27.111.195:40407
I20240502 02:10:03.992727 7554 tablet_copy_service.cc:140] P defc209b6ddf4b719d89059a990ecb05: Received BeginTabletCopySession request for tablet c7c003a62d744c138c6ef685d826b7aa from peer ca5956dd8d174df99a28e38e8a7a2524 ({username='slave'} at 127.27.111.194:57331)
I20240502 02:10:03.992832 7554 tablet_copy_service.cc:161] P defc209b6ddf4b719d89059a990ecb05: Beginning new tablet copy session on tablet c7c003a62d744c138c6ef685d826b7aa from peer ca5956dd8d174df99a28e38e8a7a2524 at {username='slave'} at 127.27.111.194:57331: session id = ca5956dd8d174df99a28e38e8a7a2524-c7c003a62d744c138c6ef685d826b7aa
I20240502 02:10:03.993578 7554 tablet_copy_source_session.cc:205] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:10:03.994522 9219 ts_tablet_manager.cc:1875] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:10:03.996580 9219 ts_tablet_manager.cc:1888] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 02:10:03.996739 9219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7c003a62d744c138c6ef685d826b7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:03.997730 9219 tablet_copy_client.cc:783] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: tablet copy: Starting download of 0 data blocks...
I20240502 02:10:03.997887 9219 tablet_copy_client.cc:647] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: tablet copy: Starting download of 1 WAL segments...
I20240502 02:10:03.999315 9219 tablet_copy_client.cc:523] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:10:04.000707 9219 tablet_bootstrap.cc:492] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: Bootstrap starting.
I20240502 02:10:04.020179 9219 tablet_bootstrap.cc:492] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:10:04.020524 9219 tablet_bootstrap.cc:492] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: Bootstrap complete.
I20240502 02:10:04.020711 9219 ts_tablet_manager.cc:1367] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20240502 02:10:04.021181 9219 raft_consensus.cc:348] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: NON_VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: true } }
I20240502 02:10:04.021322 9219 raft_consensus.cc:729] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca5956dd8d174df99a28e38e8a7a2524, State: Initialized, Role: LEARNER
I20240502 02:10:04.021428 9219 consensus_queue.cc:260] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: NON_VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: true } }
I20240502 02:10:04.021687 9219 ts_tablet_manager.cc:1398] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:10:04.022328 7554 tablet_copy_service.cc:342] P defc209b6ddf4b719d89059a990ecb05: Request end of tablet copy session ca5956dd8d174df99a28e38e8a7a2524-c7c003a62d744c138c6ef685d826b7aa received from {username='slave'} at 127.27.111.194:57331
I20240502 02:10:04.022450 7554 tablet_copy_service.cc:433] P defc209b6ddf4b719d89059a990ecb05: ending tablet copy session ca5956dd8d174df99a28e38e8a7a2524-c7c003a62d744c138c6ef685d826b7aa on tablet c7c003a62d744c138c6ef685d826b7aa with peer ca5956dd8d174df99a28e38e8a7a2524
I20240502 02:10:04.348503 7268 raft_consensus.cc:1203] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 02:10:04.474047 7401 raft_consensus.cc:1203] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 02:10:04.838908 9237 raft_consensus.cc:1050] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05: attempting to promote NON_VOTER 100889921252449596fc397b10fe903d to VOTER
I20240502 02:10:04.839309 9237 consensus_queue.cc:237] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: false } }
I20240502 02:10:04.840639 7268 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 LEARNER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:10:04.840639 8168 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:10:04.840991 9078 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:10:04.841164 9237 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:10:04.841498 7401 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:10:04.841815 9078 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:10:04.843467 9078 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 100889921252449596fc397b10fe903d (127.27.111.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: false } } }
I20240502 02:10:04.843914 7268 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 100889921252449596fc397b10fe903d (127.27.111.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: false } } }
I20240502 02:10:04.845324 7401 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 100889921252449596fc397b10fe903d (127.27.111.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: false } } }
I20240502 02:10:04.845868 8168 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 100889921252449596fc397b10fe903d (127.27.111.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: false } } }
I20240502 02:10:04.846094 7116 catalog_manager.cc:5484] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 reported cstate change: config changed from index 7 to 8, 100889921252449596fc397b10fe903d (127.27.111.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:10:04.851203 7533 consensus_queue.cc:237] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: false } }
I20240502 02:10:04.852277 7268 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:10:04.852322 8168 raft_consensus.cc:1261] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:10:04.852656 9236 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:10:04.852835 9078 consensus_queue.cc:1035] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:10:04.854336 9236 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: false } } }
I20240502 02:10:04.854650 7268 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: false } } }
I20240502 02:10:04.854703 8168 raft_consensus.cc:2932] T 2ec2a56deaf64463aa75024fb5fa32f4 P 85e53716ea13433289fe6435b6693a5b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: false } } }
I20240502 02:10:04.855969 7109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ec2a56deaf64463aa75024fb5fa32f4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:10:04.856923 7124 catalog_manager.cc:5484] T 2ec2a56deaf64463aa75024fb5fa32f4 P defc209b6ddf4b719d89059a990ecb05 reported cstate change: config changed from index 8 to 9, VOTER ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) evicted. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85e53716ea13433289fe6435b6693a5b" member_type: VOTER last_known_addr { host: "127.27.111.197" port: 36289 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:10:04.859352 7381 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ec2a56deaf64463aa75024fb5fa32f4 with delete_type TABLET_DATA_TOMBSTONED (TS ca5956dd8d174df99a28e38e8a7a2524 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44538
I20240502 02:10:04.859758 9278 tablet_replica.cc:330] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524: stopping tablet replica
I20240502 02:10:04.859916 9278 raft_consensus.cc:2229] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:10:04.860150 9278 raft_consensus.cc:2258] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:10:04.860821 9278 ts_tablet_manager.cc:1875] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:10:04.862706 9278 ts_tablet_manager.cc:1888] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 02:10:04.862789 9278 log.cc:1196] T 2ec2a56deaf64463aa75024fb5fa32f4 P ca5956dd8d174df99a28e38e8a7a2524: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/wals/2ec2a56deaf64463aa75024fb5fa32f4
I20240502 02:10:04.863376 7108 catalog_manager.cc:4830] TS ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194:40893): tablet 2ec2a56deaf64463aa75024fb5fa32f4 (table rebalance_test_table_0 [id=424baa04ef41432ebdec6d07ed41cd5e]) successfully deleted
I20240502 02:10:04.868010 9237 raft_consensus.cc:1050] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05: attempting to promote NON_VOTER ca5956dd8d174df99a28e38e8a7a2524 to VOTER
I20240502 02:10:04.868397 9237 consensus_queue.cc:237] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: false } }
I20240502 02:10:04.869985 8299 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:10:04.870010 7268 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:10:04.870023 7401 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 LEARNER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:10:04.870432 9237 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:10:04.870640 9078 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:10:04.870777 9280 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:10:04.872613 9078 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: false } } }
I20240502 02:10:04.873008 8299 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: false } } }
I20240502 02:10:04.873769 7268 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: false } } }
I20240502 02:10:04.875411 7401 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: false } } }
I20240502 02:10:04.875622 7120 catalog_manager.cc:5484] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da reported cstate change: config changed from index 7 to 8, ca5956dd8d174df99a28e38e8a7a2524 (127.27.111.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "100889921252449596fc397b10fe903d" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 43115 } attrs { replace: true } } peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: false } } }
I20240502 02:10:04.879050 7533 consensus_queue.cc:237] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: false } }
I20240502 02:10:04.880122 8299 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:10:04.880466 7401 raft_consensus.cc:1261] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Refusing update from remote peer defc209b6ddf4b719d89059a990ecb05: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:10:04.880465 9237 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:10:04.880811 9237 consensus_queue.cc:1035] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:10:04.881958 9280 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P defc209b6ddf4b719d89059a990ecb05 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: false } } }
I20240502 02:10:04.882627 7401 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: false } } }
I20240502 02:10:04.883512 7109 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c7c003a62d744c138c6ef685d826b7aa with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:10:04.883718 8299 raft_consensus.cc:2932] T c7c003a62d744c138c6ef685d826b7aa P efabf88707da4c13b7ec85bbfd8a89da [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: false } } }
I20240502 02:10:04.885061 7124 catalog_manager.cc:5484] T c7c003a62d744c138c6ef685d826b7aa P ca5956dd8d174df99a28e38e8a7a2524 reported cstate change: config changed from index 8 to 9, VOTER 100889921252449596fc397b10fe903d (127.27.111.193) evicted. New cstate: current_term: 1 leader_uuid: "defc209b6ddf4b719d89059a990ecb05" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "defc209b6ddf4b719d89059a990ecb05" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 40407 } } peers { permanent_uuid: "efabf88707da4c13b7ec85bbfd8a89da" member_type: VOTER last_known_addr { host: "127.27.111.196" port: 45431 } attrs { promote: false } } peers { permanent_uuid: "ca5956dd8d174df99a28e38e8a7a2524" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 40893 } attrs { promote: false } } }
I20240502 02:10:04.887576 7247 tablet_service.cc:1514] Processing DeleteTablet for tablet c7c003a62d744c138c6ef685d826b7aa with delete_type TABLET_DATA_TOMBSTONED (TS 100889921252449596fc397b10fe903d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44892
I20240502 02:10:04.887952 9285 tablet_replica.cc:330] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d: stopping tablet replica
I20240502 02:10:04.888095 9285 raft_consensus.cc:2229] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:10:04.888288 9285 raft_consensus.cc:2258] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:10:04.888886 9285 ts_tablet_manager.cc:1875] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:10:04.890728 9285 ts_tablet_manager.cc:1888] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 02:10:04.890798 9285 log.cc:1196] T c7c003a62d744c138c6ef685d826b7aa P 100889921252449596fc397b10fe903d: Deleting WAL directory at /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/wals/c7c003a62d744c138c6ef685d826b7aa
I20240502 02:10:04.891377 7110 catalog_manager.cc:4830] TS 100889921252449596fc397b10fe903d (127.27.111.193:43115): tablet c7c003a62d744c138c6ef685d826b7aa (table rebalance_test_table_1 [id=789c28d35c724bb8a67e0674bacf590b]) successfully deleted
I20240502 02:10:05.051934 7247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:10:05.052269 7511 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:05.053203 7381 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:05.056210 8279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:10:05.064751 8148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:10:05.140136 7381 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:05.140669 8148 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:10:05.141666 8279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:10:05.143885 7511 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:05.146557 7247 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:10:05.186057 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 7185
I20240502 02:10:05.193732 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 7318
I20240502 02:10:05.201362 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 7450
I20240502 02:10:05.208686 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 8217
I20240502 02:10:05.214944 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 8086
I20240502 02:10:05.221225 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 7093
2024-05-02T02:10:05Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2 (13411 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1
I20240502 02:10:05.251260 28095 test_util.cc:268] Using random seed: 436178613
I20240502 02:10:05.252113 28095 ts_itest-base.cc:112] Starting cluster with:
I20240502 02:10:05.252161 28095 ts_itest-base.cc:113] --------------
I20240502 02:10:05.252173 28095 ts_itest-base.cc:114] 4 tablet servers
I20240502 02:10:05.252183 28095 ts_itest-base.cc:115] 3 replicas per TS
I20240502 02:10:05.252194 28095 ts_itest-base.cc:116] --------------
2024-05-02T02:10:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T02:10:05Z Disabled control of system clock
I20240502 02:10:05.260700 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.111.254:39465
--webserver_interface=127.27.111.254
--webserver_port=0
--builtin_ntp_servers=127.27.111.212:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.111.254:39465
--location_mapping_cmd=/tmp/dist-test-task43FDSt/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 02:10:05.374027 9363 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 02:10:05.374338 9363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:10:05.374418 9363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:10:05.374490 9363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:10:05.377593 9363 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:10:05.377691 9363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 02:10:05.377740 9363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:10:05.377775 9363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 02:10:05.377818 9363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 02:10:05.381634 9363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task43FDSt/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--ipki_ca_key_size=768
--master_addresses=127.27.111.254:39465
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.111.254:39465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.27.111.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.9363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:10:05.382665 9363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:10:05.383842 9363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 02:10:05.391922 9363 server_base.cc:1034] running on GCE node
W20240502 02:10:05.392406 9372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:10:05.393085 9368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:10:05.393383 9369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:10:05.393908 9363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:10:05.394941 9363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:10:05.396102 9363 hybrid_clock.cc:648] HybridClock initialized: now 1714615805396089 us; error 31 us; skew 500 ppm
I20240502 02:10:05.398078 9363 webserver.cc:460] Webserver started at http://127.27.111.254:37853/ using document root <none> and password file <none>
I20240502 02:10:05.398697 9363 fs_manager.cc:359] Metadata directory not provided
I20240502 02:10:05.398762 9363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:10:05.398959 9363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:10:05.400697 9363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "db31820b2cd945bca69d235f39a51ab7"
format_stamp: "Formatted at 2024-05-02 02:10:05 on dist-test-slave-0pw1"
I20240502 02:10:05.401172 9363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "db31820b2cd945bca69d235f39a51ab7"
format_stamp: "Formatted at 2024-05-02 02:10:05 on dist-test-slave-0pw1"
I20240502 02:10:05.404567 9363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 02:10:05.406831 9378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:05.407888 9363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 02:10:05.408002 9363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
uuid: "db31820b2cd945bca69d235f39a51ab7"
format_stamp: "Formatted at 2024-05-02 02:10:05 on dist-test-slave-0pw1"
I20240502 02:10:05.408110 9363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:10:05.424352 9363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:10:05.425082 9363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:10:05.425251 9363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:10:05.432173 9363 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.254:39465
I20240502 02:10:05.432183 9431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.254:39465 every 8 connection(s)
I20240502 02:10:05.433178 9363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 02:10:05.435894 9432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:05.437006 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 9363
I20240502 02:10:05.437160 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 02:10:05.441982 9432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7: Bootstrap starting.
I20240502 02:10:05.444547 9432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:05.445410 9432 log.cc:824] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7: Log is configured to *not* fsync() on all Append() calls
I20240502 02:10:05.447289 9432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7: No bootstrap required, opened a new log
I20240502 02:10:05.449841 9432 raft_consensus.cc:348] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db31820b2cd945bca69d235f39a51ab7" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 39465 } }
I20240502 02:10:05.450026 9432 raft_consensus.cc:374] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:05.450080 9432 raft_consensus.cc:729] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db31820b2cd945bca69d235f39a51ab7, State: Initialized, Role: FOLLOWER
I20240502 02:10:05.450536 9432 consensus_queue.cc:260] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db31820b2cd945bca69d235f39a51ab7" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 39465 } }
I20240502 02:10:05.450666 9432 raft_consensus.cc:388] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:10:05.450727 9432 raft_consensus.cc:482] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:10:05.450832 9432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:05.451632 9432 raft_consensus.cc:504] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db31820b2cd945bca69d235f39a51ab7" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 39465 } }
I20240502 02:10:05.451943 9432 leader_election.cc:302] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db31820b2cd945bca69d235f39a51ab7; no voters:
I20240502 02:10:05.452214 9432 leader_election.cc:288] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 02:10:05.452741 9437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:10:05.452961 9437 raft_consensus.cc:686] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [term 1 LEADER]: Becoming Leader. State: Replica: db31820b2cd945bca69d235f39a51ab7, State: Running, Role: LEADER
I20240502 02:10:05.453305 9437 consensus_queue.cc:237] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db31820b2cd945bca69d235f39a51ab7" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 39465 } }
I20240502 02:10:05.453373 9432 sys_catalog.cc:564] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 02:10:05.454866 9439 sys_catalog.cc:455] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db31820b2cd945bca69d235f39a51ab7. Latest consensus state: current_term: 1 leader_uuid: "db31820b2cd945bca69d235f39a51ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db31820b2cd945bca69d235f39a51ab7" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 39465 } } }
I20240502 02:10:05.454843 9438 sys_catalog.cc:455] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db31820b2cd945bca69d235f39a51ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db31820b2cd945bca69d235f39a51ab7" member_type: VOTER last_known_addr { host: "127.27.111.254" port: 39465 } } }
I20240502 02:10:05.455008 9439 sys_catalog.cc:458] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [sys.catalog]: This master's current role is: LEADER
I20240502 02:10:05.455015 9438 sys_catalog.cc:458] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7 [sys.catalog]: This master's current role is: LEADER
I20240502 02:10:05.455418 9449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 02:10:05.458196 9449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 02:10:05.463070 9449 catalog_manager.cc:1347] Generated new cluster ID: 51f93c5c216e496c8ac0ce679da06101
I20240502 02:10:05.463148 9449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 02:10:05.479851 9449 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 02:10:05.481148 9449 catalog_manager.cc:1504] Loading token signing keys...
I20240502 02:10:05.491025 9449 catalog_manager.cc:5857] T 00000000000000000000000000000000 P db31820b2cd945bca69d235f39a51ab7: Generated new TSK 0
I20240502 02:10:05.491828 9449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 02:10:05.502329 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.193:0
--local_ip_for_outbound_sockets=127.27.111.193
--webserver_interface=127.27.111.193
--webserver_port=0
--tserver_master_addrs=127.27.111.254:39465
--builtin_ntp_servers=127.27.111.212:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 02:10:05.618480 9456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:10:05.618721 9456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:10:05.618794 9456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:10:05.621984 9456 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:10:05.622153 9456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:10:05.622319 9456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.193
I20240502 02:10:05.626255 9456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.27.111.193
--webserver_port=0
--tserver_master_addrs=127.27.111.254:39465
--never_fsync=true
--heap_profile_path=/tmp/kudu.9456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.193
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:10:05.627241 9456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:10:05.628445 9456 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:10:05.637245 9462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:10:05.637423 9461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:10:05.638186 9464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:10:05.638782 9456 server_base.cc:1034] running on GCE node
I20240502 02:10:05.639225 9456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:10:05.639873 9456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:10:05.641041 9456 hybrid_clock.cc:648] HybridClock initialized: now 1714615805641017 us; error 37 us; skew 500 ppm
I20240502 02:10:05.643258 9456 webserver.cc:460] Webserver started at http://127.27.111.193:35791/ using document root <none> and password file <none>
I20240502 02:10:05.643864 9456 fs_manager.cc:359] Metadata directory not provided
I20240502 02:10:05.643931 9456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:10:05.644125 9456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:10:05.645898 9456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6a1bc5b598674f24bbaf0714130bcf5b"
format_stamp: "Formatted at 2024-05-02 02:10:05 on dist-test-slave-0pw1"
I20240502 02:10:05.646436 9456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6a1bc5b598674f24bbaf0714130bcf5b"
format_stamp: "Formatted at 2024-05-02 02:10:05 on dist-test-slave-0pw1"
I20240502 02:10:05.649955 9456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 02:10:05.652285 9470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:05.653318 9456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 02:10:05.653488 9456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6a1bc5b598674f24bbaf0714130bcf5b"
format_stamp: "Formatted at 2024-05-02 02:10:05 on dist-test-slave-0pw1"
I20240502 02:10:05.653609 9456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:10:05.679105 9456 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:10:05.679760 9456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:10:05.679937 9456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:10:05.680495 9456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:10:05.681411 9456 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:10:05.681459 9456 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:05.681519 9456 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:10:05.681545 9456 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:05.690066 9456 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.193:37619
I20240502 02:10:05.690091 9585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.193:37619 every 8 connection(s)
I20240502 02:10:05.691033 9456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 02:10:05.699388 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 9456
I20240502 02:10:05.699545 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 02:10:05.699877 9586 heartbeater.cc:347] Connected to a master server at 127.27.111.254:39465
I20240502 02:10:05.700129 9586 heartbeater.cc:464] Registering TS with master...
I20240502 02:10:05.700659 9586 heartbeater.cc:511] Master 127.27.111.254:39465 requested a full tablet report, sending...
I20240502 02:10:05.701690 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.194:0
--local_ip_for_outbound_sockets=127.27.111.194
--webserver_interface=127.27.111.194
--webserver_port=0
--tserver_master_addrs=127.27.111.254:39465
--builtin_ntp_servers=127.27.111.212:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:10:05.739857 9395 ts_manager.cc:194] Registered new tserver with Master: 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619)
I20240502 02:10:05.741230 9395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.193:46111
W20240502 02:10:05.815128 9589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:10:05.815387 9589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:10:05.815433 9589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:10:05.818625 9589 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:10:05.818706 9589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:10:05.818795 9589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.194
I20240502 02:10:05.822621 9589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.27.111.194
--webserver_port=0
--tserver_master_addrs=127.27.111.254:39465
--never_fsync=true
--heap_profile_path=/tmp/kudu.9589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.194
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:10:05.823654 9589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:10:05.824819 9589 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:10:05.833276 9598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:10:05.833940 9595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:10:05.834626 9596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:10:05.835307 9589 server_base.cc:1034] running on GCE node
I20240502 02:10:05.835729 9589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:10:05.836347 9589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:10:05.837518 9589 hybrid_clock.cc:648] HybridClock initialized: now 1714615805837494 us; error 32 us; skew 500 ppm
I20240502 02:10:05.839627 9589 webserver.cc:460] Webserver started at http://127.27.111.194:44833/ using document root <none> and password file <none>
I20240502 02:10:05.840219 9589 fs_manager.cc:359] Metadata directory not provided
I20240502 02:10:05.840286 9589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:10:05.840497 9589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:10:05.842245 9589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f74e6d70134a4212a44a0da8100aeb35"
format_stamp: "Formatted at 2024-05-02 02:10:05 on dist-test-slave-0pw1"
I20240502 02:10:05.842792 9589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f74e6d70134a4212a44a0da8100aeb35"
format_stamp: "Formatted at 2024-05-02 02:10:05 on dist-test-slave-0pw1"
I20240502 02:10:05.846321 9589 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 02:10:05.848748 9604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:05.849756 9589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 02:10:05.849870 9589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f74e6d70134a4212a44a0da8100aeb35"
format_stamp: "Formatted at 2024-05-02 02:10:05 on dist-test-slave-0pw1"
I20240502 02:10:05.849963 9589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:10:05.867195 9589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:10:05.867897 9589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:10:05.868065 9589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:10:05.868624 9589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:10:05.869525 9589 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:10:05.869573 9589 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:05.869633 9589 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:10:05.869658 9589 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:05.878371 9589 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.194:36383
I20240502 02:10:05.878443 9718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.194:36383 every 8 connection(s)
I20240502 02:10:05.879360 9589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 02:10:05.888085 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 9589
I20240502 02:10:05.888083 9719 heartbeater.cc:347] Connected to a master server at 127.27.111.254:39465
I20240502 02:10:05.888217 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 02:10:05.888307 9719 heartbeater.cc:464] Registering TS with master...
I20240502 02:10:05.888872 9719 heartbeater.cc:511] Master 127.27.111.254:39465 requested a full tablet report, sending...
I20240502 02:10:05.890507 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.195:0
--local_ip_for_outbound_sockets=127.27.111.195
--webserver_interface=127.27.111.195
--webserver_port=0
--tserver_master_addrs=127.27.111.254:39465
--builtin_ntp_servers=127.27.111.212:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:10:05.924795 9395 ts_manager.cc:194] Registered new tserver with Master: f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194:36383)
I20240502 02:10:05.925742 9395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.194:58771
W20240502 02:10:06.001955 9722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:10:06.002195 9722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:10:06.002241 9722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:10:06.005312 9722 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:10:06.005383 9722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:10:06.005453 9722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.195
I20240502 02:10:06.009133 9722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.27.111.195
--webserver_port=0
--tserver_master_addrs=127.27.111.254:39465
--never_fsync=true
--heap_profile_path=/tmp/kudu.9722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.195
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:10:06.010066 9722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:10:06.011237 9722 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:10:06.020148 9731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:10:06.020437 9728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:10:06.021090 9729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:10:06.021756 9722 server_base.cc:1034] running on GCE node
I20240502 02:10:06.022143 9722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:10:06.022702 9722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:10:06.023850 9722 hybrid_clock.cc:648] HybridClock initialized: now 1714615806023841 us; error 28 us; skew 500 ppm
I20240502 02:10:06.025703 9722 webserver.cc:460] Webserver started at http://127.27.111.195:40995/ using document root <none> and password file <none>
I20240502 02:10:06.026258 9722 fs_manager.cc:359] Metadata directory not provided
I20240502 02:10:06.026319 9722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:10:06.026540 9722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:10:06.028167 9722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8736c9ad53e8442fad4b8a577791aff8"
format_stamp: "Formatted at 2024-05-02 02:10:06 on dist-test-slave-0pw1"
I20240502 02:10:06.028633 9722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8736c9ad53e8442fad4b8a577791aff8"
format_stamp: "Formatted at 2024-05-02 02:10:06 on dist-test-slave-0pw1"
I20240502 02:10:06.031838 9722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 02:10:06.033912 9737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:06.034971 9722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 02:10:06.035079 9722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8736c9ad53e8442fad4b8a577791aff8"
format_stamp: "Formatted at 2024-05-02 02:10:06 on dist-test-slave-0pw1"
I20240502 02:10:06.035154 9722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:10:06.052178 9722 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:10:06.052855 9722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:10:06.053002 9722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:10:06.053578 9722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:10:06.054657 9722 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:10:06.054714 9722 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:06.054772 9722 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:10:06.054802 9722 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:06.063133 9722 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.195:35865
I20240502 02:10:06.063159 9853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.195:35865 every 8 connection(s)
I20240502 02:10:06.064175 9722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 02:10:06.066963 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 9722
I20240502 02:10:06.067097 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 02:10:06.069363 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.196:0
--local_ip_for_outbound_sockets=127.27.111.196
--webserver_interface=127.27.111.196
--webserver_port=0
--tserver_master_addrs=127.27.111.254:39465
--builtin_ntp_servers=127.27.111.212:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:10:06.074026 9854 heartbeater.cc:347] Connected to a master server at 127.27.111.254:39465
I20240502 02:10:06.074249 9854 heartbeater.cc:464] Registering TS with master...
I20240502 02:10:06.074842 9854 heartbeater.cc:511] Master 127.27.111.254:39465 requested a full tablet report, sending...
I20240502 02:10:06.108812 9395 ts_manager.cc:194] Registered new tserver with Master: 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195:35865)
I20240502 02:10:06.109782 9395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.195:40525
W20240502 02:10:06.180014 9857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:10:06.180274 9857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:10:06.180318 9857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:10:06.183545 9857 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:10:06.183626 9857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:10:06.183699 9857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.196
I20240502 02:10:06.187464 9857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.111.196
--webserver_port=0
--tserver_master_addrs=127.27.111.254:39465
--never_fsync=true
--heap_profile_path=/tmp/kudu.9857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.196
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:10:06.188418 9857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:10:06.189546 9857 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:10:06.198081 9866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:10:06.198655 9863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:10:06.199213 9864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:10:06.199906 9857 server_base.cc:1034] running on GCE node
I20240502 02:10:06.200326 9857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:10:06.200937 9857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:10:06.202117 9857 hybrid_clock.cc:648] HybridClock initialized: now 1714615806202104 us; error 33 us; skew 500 ppm
I20240502 02:10:06.204277 9857 webserver.cc:460] Webserver started at http://127.27.111.196:42551/ using document root <none> and password file <none>
I20240502 02:10:06.204895 9857 fs_manager.cc:359] Metadata directory not provided
I20240502 02:10:06.204962 9857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:10:06.205142 9857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:10:06.206900 9857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5cb7c03f646743a0b5662b2a7529395b"
format_stamp: "Formatted at 2024-05-02 02:10:06 on dist-test-slave-0pw1"
I20240502 02:10:06.207399 9857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5cb7c03f646743a0b5662b2a7529395b"
format_stamp: "Formatted at 2024-05-02 02:10:06 on dist-test-slave-0pw1"
I20240502 02:10:06.210968 9857 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 02:10:06.213384 9872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:06.214494 9857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 02:10:06.214602 9857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5cb7c03f646743a0b5662b2a7529395b"
format_stamp: "Formatted at 2024-05-02 02:10:06 on dist-test-slave-0pw1"
I20240502 02:10:06.214704 9857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:10:06.231881 9857 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:10:06.232564 9857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:10:06.232729 9857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:10:06.233291 9857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:10:06.234241 9857 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:10:06.234292 9857 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:06.234355 9857 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:10:06.234411 9857 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:06.242990 9857 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.196:44107
I20240502 02:10:06.243010 9988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.196:44107 every 8 connection(s)
I20240502 02:10:06.243978 9857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 02:10:06.245143 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 9857
I20240502 02:10:06.245280 28095 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 02:10:06.253010 9989 heartbeater.cc:347] Connected to a master server at 127.27.111.254:39465
I20240502 02:10:06.253248 9989 heartbeater.cc:464] Registering TS with master...
I20240502 02:10:06.253798 9989 heartbeater.cc:511] Master 127.27.111.254:39465 requested a full tablet report, sending...
I20240502 02:10:06.289037 9395 ts_manager.cc:194] Registered new tserver with Master: 5cb7c03f646743a0b5662b2a7529395b (127.27.111.196:44107)
I20240502 02:10:06.289775 9395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.196:50081
I20240502 02:10:06.295675 28095 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 02:10:06.305155 28095 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 02:10:06.305236 28095 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task43FDSt/build/debug/bin/kudu with pid 9857
I20240502 02:10:06.743786 9586 heartbeater.cc:503] Master 127.27.111.254:39465 was elected leader, sending a full tablet report...
I20240502 02:10:06.928459 9719 heartbeater.cc:503] Master 127.27.111.254:39465 was elected leader, sending a full tablet report...
I20240502 02:10:07.112263 9854 heartbeater.cc:503] Master 127.27.111.254:39465 was elected leader, sending a full tablet report...
I20240502 02:10:10.062853 9395 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43074:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 02:10:10.066884 9395 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 02:10:10.085954 9653 tablet_service.cc:1467] Processing CreateTablet for tablet fb12390841324803a9b0e256a82735ad (DEFAULT_TABLE table=rebalance_test_table_0 [id=044a784d58474a3992363549fca1b546]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.085950 9784 tablet_service.cc:1467] Processing CreateTablet for tablet 3c86c5791e074d3cae477c8d67077778 (DEFAULT_TABLE table=rebalance_test_table_0 [id=044a784d58474a3992363549fca1b546]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.085953 9783 tablet_service.cc:1467] Processing CreateTablet for tablet 8c63071e4946459fb633c2b8e8a6be78 (DEFAULT_TABLE table=rebalance_test_table_0 [id=044a784d58474a3992363549fca1b546]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.086508 9650 tablet_service.cc:1467] Processing CreateTablet for tablet 8c63071e4946459fb633c2b8e8a6be78 (DEFAULT_TABLE table=rebalance_test_table_0 [id=044a784d58474a3992363549fca1b546]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.086089 9652 tablet_service.cc:1467] Processing CreateTablet for tablet 3c86c5791e074d3cae477c8d67077778 (DEFAULT_TABLE table=rebalance_test_table_0 [id=044a784d58474a3992363549fca1b546]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.087126 9650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c63071e4946459fb633c2b8e8a6be78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.087131 9783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c63071e4946459fb633c2b8e8a6be78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.087687 9784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c86c5791e074d3cae477c8d67077778. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.088891 9517 tablet_service.cc:1467] Processing CreateTablet for tablet 8c63071e4946459fb633c2b8e8a6be78 (DEFAULT_TABLE table=rebalance_test_table_0 [id=044a784d58474a3992363549fca1b546]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.088893 9519 tablet_service.cc:1467] Processing CreateTablet for tablet fb12390841324803a9b0e256a82735ad (DEFAULT_TABLE table=rebalance_test_table_0 [id=044a784d58474a3992363549fca1b546]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.089844 9517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c63071e4946459fb633c2b8e8a6be78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.090468 9519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb12390841324803a9b0e256a82735ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.091085 9518 tablet_service.cc:1467] Processing CreateTablet for tablet 3c86c5791e074d3cae477c8d67077778 (DEFAULT_TABLE table=rebalance_test_table_0 [id=044a784d58474a3992363549fca1b546]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.091337 9518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c86c5791e074d3cae477c8d67077778. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.091814 9785 tablet_service.cc:1467] Processing CreateTablet for tablet fb12390841324803a9b0e256a82735ad (DEFAULT_TABLE table=rebalance_test_table_0 [id=044a784d58474a3992363549fca1b546]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.092080 9785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb12390841324803a9b0e256a82735ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.094162 9652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c86c5791e074d3cae477c8d67077778. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.094771 10009 tablet_bootstrap.cc:492] T 8c63071e4946459fb633c2b8e8a6be78 P 8736c9ad53e8442fad4b8a577791aff8: Bootstrap starting.
I20240502 02:10:10.096460 9653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb12390841324803a9b0e256a82735ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.097081 10009 tablet_bootstrap.cc:654] T 8c63071e4946459fb633c2b8e8a6be78 P 8736c9ad53e8442fad4b8a577791aff8: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.098026 10009 log.cc:824] T 8c63071e4946459fb633c2b8e8a6be78 P 8736c9ad53e8442fad4b8a577791aff8: Log is configured to *not* fsync() on all Append() calls
I20240502 02:10:10.098780 10010 tablet_bootstrap.cc:492] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b: Bootstrap starting.
I20240502 02:10:10.098903 10008 tablet_bootstrap.cc:492] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35: Bootstrap starting.
I20240502 02:10:10.100596 10009 tablet_bootstrap.cc:492] T 8c63071e4946459fb633c2b8e8a6be78 P 8736c9ad53e8442fad4b8a577791aff8: No bootstrap required, opened a new log
I20240502 02:10:10.100734 10008 tablet_bootstrap.cc:654] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.100839 10009 ts_tablet_manager.cc:1367] T 8c63071e4946459fb633c2b8e8a6be78 P 8736c9ad53e8442fad4b8a577791aff8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 02:10:10.101256 10010 tablet_bootstrap.cc:654] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.101497 10008 log.cc:824] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35: Log is configured to *not* fsync() on all Append() calls
I20240502 02:10:10.102010 10010 log.cc:824] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b: Log is configured to *not* fsync() on all Append() calls
I20240502 02:10:10.103085 10008 tablet_bootstrap.cc:492] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35: No bootstrap required, opened a new log
I20240502 02:10:10.103257 10008 ts_tablet_manager.cc:1367] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 02:10:10.103721 10010 tablet_bootstrap.cc:492] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b: No bootstrap required, opened a new log
I20240502 02:10:10.103890 10010 ts_tablet_manager.cc:1367] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 02:10:10.104444 10009 raft_consensus.cc:348] T 8c63071e4946459fb633c2b8e8a6be78 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.104672 10009 raft_consensus.cc:374] T 8c63071e4946459fb633c2b8e8a6be78 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.104773 10009 raft_consensus.cc:729] T 8c63071e4946459fb633c2b8e8a6be78 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8736c9ad53e8442fad4b8a577791aff8, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.105337 10009 consensus_queue.cc:260] T 8c63071e4946459fb633c2b8e8a6be78 P 8736c9ad53e8442fad4b8a577791aff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.106251 10009 ts_tablet_manager.cc:1398] T 8c63071e4946459fb633c2b8e8a6be78 P 8736c9ad53e8442fad4b8a577791aff8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 02:10:10.106516 10008 raft_consensus.cc:348] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.106720 10008 raft_consensus.cc:374] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.106803 10008 raft_consensus.cc:729] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6d70134a4212a44a0da8100aeb35, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.107241 10009 tablet_bootstrap.cc:492] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8: Bootstrap starting.
I20240502 02:10:10.107338 10008 consensus_queue.cc:260] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.107542 10010 raft_consensus.cc:348] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.107793 10010 raft_consensus.cc:374] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.107880 10010 raft_consensus.cc:729] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1bc5b598674f24bbaf0714130bcf5b, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.108261 10008 ts_tablet_manager.cc:1398] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 02:10:10.108384 10010 consensus_queue.cc:260] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.108631 10008 tablet_bootstrap.cc:492] T 3c86c5791e074d3cae477c8d67077778 P f74e6d70134a4212a44a0da8100aeb35: Bootstrap starting.
I20240502 02:10:10.108945 10009 tablet_bootstrap.cc:654] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.109376 10010 ts_tablet_manager.cc:1398] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20240502 02:10:10.109685 10008 tablet_bootstrap.cc:654] T 3c86c5791e074d3cae477c8d67077778 P f74e6d70134a4212a44a0da8100aeb35: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.110807 10008 tablet_bootstrap.cc:492] T 3c86c5791e074d3cae477c8d67077778 P f74e6d70134a4212a44a0da8100aeb35: No bootstrap required, opened a new log
I20240502 02:10:10.110962 10008 ts_tablet_manager.cc:1367] T 3c86c5791e074d3cae477c8d67077778 P f74e6d70134a4212a44a0da8100aeb35: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 02:10:10.111493 10008 raft_consensus.cc:348] T 3c86c5791e074d3cae477c8d67077778 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.111613 10008 raft_consensus.cc:374] T 3c86c5791e074d3cae477c8d67077778 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.111672 10008 raft_consensus.cc:729] T 3c86c5791e074d3cae477c8d67077778 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6d70134a4212a44a0da8100aeb35, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.111893 10008 consensus_queue.cc:260] T 3c86c5791e074d3cae477c8d67077778 P f74e6d70134a4212a44a0da8100aeb35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.111940 10010 tablet_bootstrap.cc:492] T fb12390841324803a9b0e256a82735ad P 6a1bc5b598674f24bbaf0714130bcf5b: Bootstrap starting.
I20240502 02:10:10.112267 10008 ts_tablet_manager.cc:1398] T 3c86c5791e074d3cae477c8d67077778 P f74e6d70134a4212a44a0da8100aeb35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:10:10.112442 10008 tablet_bootstrap.cc:492] T fb12390841324803a9b0e256a82735ad P f74e6d70134a4212a44a0da8100aeb35: Bootstrap starting.
I20240502 02:10:10.112980 10010 tablet_bootstrap.cc:654] T fb12390841324803a9b0e256a82735ad P 6a1bc5b598674f24bbaf0714130bcf5b: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.113301 10008 tablet_bootstrap.cc:654] T fb12390841324803a9b0e256a82735ad P f74e6d70134a4212a44a0da8100aeb35: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.114517 10009 tablet_bootstrap.cc:492] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8: No bootstrap required, opened a new log
I20240502 02:10:10.114625 10009 ts_tablet_manager.cc:1367] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240502 02:10:10.114624 10010 tablet_bootstrap.cc:492] T fb12390841324803a9b0e256a82735ad P 6a1bc5b598674f24bbaf0714130bcf5b: No bootstrap required, opened a new log
I20240502 02:10:10.115247 10009 raft_consensus.cc:348] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } }
I20240502 02:10:10.115343 10009 raft_consensus.cc:374] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.115370 10009 raft_consensus.cc:729] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8736c9ad53e8442fad4b8a577791aff8, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.114706 10010 ts_tablet_manager.cc:1367] T fb12390841324803a9b0e256a82735ad P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 02:10:10.115515 10009 consensus_queue.cc:260] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } }
I20240502 02:10:10.115667 10008 tablet_bootstrap.cc:492] T fb12390841324803a9b0e256a82735ad P f74e6d70134a4212a44a0da8100aeb35: No bootstrap required, opened a new log
I20240502 02:10:10.115742 10008 ts_tablet_manager.cc:1367] T fb12390841324803a9b0e256a82735ad P f74e6d70134a4212a44a0da8100aeb35: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 02:10:10.115795 10009 ts_tablet_manager.cc:1398] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:10:10.115968 10009 tablet_bootstrap.cc:492] T 3c86c5791e074d3cae477c8d67077778 P 8736c9ad53e8442fad4b8a577791aff8: Bootstrap starting.
I20240502 02:10:10.115975 10010 raft_consensus.cc:348] T fb12390841324803a9b0e256a82735ad P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } }
I20240502 02:10:10.116075 10010 raft_consensus.cc:374] T fb12390841324803a9b0e256a82735ad P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.116111 10010 raft_consensus.cc:729] T fb12390841324803a9b0e256a82735ad P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1bc5b598674f24bbaf0714130bcf5b, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.116175 10008 raft_consensus.cc:348] T fb12390841324803a9b0e256a82735ad P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } }
I20240502 02:10:10.116264 10008 raft_consensus.cc:374] T fb12390841324803a9b0e256a82735ad P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.116233 10010 consensus_queue.cc:260] T fb12390841324803a9b0e256a82735ad P 6a1bc5b598674f24bbaf0714130bcf5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } }
I20240502 02:10:10.116293 10008 raft_consensus.cc:729] T fb12390841324803a9b0e256a82735ad P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6d70134a4212a44a0da8100aeb35, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.116400 10008 consensus_queue.cc:260] T fb12390841324803a9b0e256a82735ad P f74e6d70134a4212a44a0da8100aeb35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } }
I20240502 02:10:10.116513 10010 ts_tablet_manager.cc:1398] T fb12390841324803a9b0e256a82735ad P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:10:10.116670 10010 tablet_bootstrap.cc:492] T 8c63071e4946459fb633c2b8e8a6be78 P 6a1bc5b598674f24bbaf0714130bcf5b: Bootstrap starting.
I20240502 02:10:10.116660 10008 ts_tablet_manager.cc:1398] T fb12390841324803a9b0e256a82735ad P f74e6d70134a4212a44a0da8100aeb35: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:10:10.116941 10009 tablet_bootstrap.cc:654] T 3c86c5791e074d3cae477c8d67077778 P 8736c9ad53e8442fad4b8a577791aff8: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.117583 10010 tablet_bootstrap.cc:654] T 8c63071e4946459fb633c2b8e8a6be78 P 6a1bc5b598674f24bbaf0714130bcf5b: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.119167 10010 tablet_bootstrap.cc:492] T 8c63071e4946459fb633c2b8e8a6be78 P 6a1bc5b598674f24bbaf0714130bcf5b: No bootstrap required, opened a new log
I20240502 02:10:10.119309 10010 ts_tablet_manager.cc:1367] T 8c63071e4946459fb633c2b8e8a6be78 P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 02:10:10.119944 10010 raft_consensus.cc:348] T 8c63071e4946459fb633c2b8e8a6be78 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.120138 10010 raft_consensus.cc:374] T 8c63071e4946459fb633c2b8e8a6be78 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.120223 10010 raft_consensus.cc:729] T 8c63071e4946459fb633c2b8e8a6be78 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1bc5b598674f24bbaf0714130bcf5b, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.120381 10010 consensus_queue.cc:260] T 8c63071e4946459fb633c2b8e8a6be78 P 6a1bc5b598674f24bbaf0714130bcf5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.120704 10010 ts_tablet_manager.cc:1398] T 8c63071e4946459fb633c2b8e8a6be78 P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:10:10.121462 10009 tablet_bootstrap.cc:492] T 3c86c5791e074d3cae477c8d67077778 P 8736c9ad53e8442fad4b8a577791aff8: No bootstrap required, opened a new log
I20240502 02:10:10.121573 10009 ts_tablet_manager.cc:1367] T 3c86c5791e074d3cae477c8d67077778 P 8736c9ad53e8442fad4b8a577791aff8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240502 02:10:10.121994 10009 raft_consensus.cc:348] T 3c86c5791e074d3cae477c8d67077778 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.122102 10009 raft_consensus.cc:374] T 3c86c5791e074d3cae477c8d67077778 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.122136 10009 raft_consensus.cc:729] T 3c86c5791e074d3cae477c8d67077778 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8736c9ad53e8442fad4b8a577791aff8, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.122282 10009 consensus_queue.cc:260] T 3c86c5791e074d3cae477c8d67077778 P 8736c9ad53e8442fad4b8a577791aff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.122568 10009 ts_tablet_manager.cc:1398] T 3c86c5791e074d3cae477c8d67077778 P 8736c9ad53e8442fad4b8a577791aff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20240502 02:10:10.134063 9720 tablet.cc:2369] T 3c86c5791e074d3cae477c8d67077778 P f74e6d70134a4212a44a0da8100aeb35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 02:10:10.167742 10016 raft_consensus.cc:482] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:10:10.167986 10016 raft_consensus.cc:504] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.169188 10016 leader_election.cc:288] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194:36383), 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195:35865)
I20240502 02:10:10.173589 9807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c86c5791e074d3cae477c8d67077778" candidate_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8736c9ad53e8442fad4b8a577791aff8" is_pre_election: true
I20240502 02:10:10.173993 9807 raft_consensus.cc:2454] T 3c86c5791e074d3cae477c8d67077778 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a1bc5b598674f24bbaf0714130bcf5b in term 0.
I20240502 02:10:10.173589 9673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c86c5791e074d3cae477c8d67077778" candidate_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6d70134a4212a44a0da8100aeb35" is_pre_election: true
I20240502 02:10:10.174417 9673 raft_consensus.cc:2454] T 3c86c5791e074d3cae477c8d67077778 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a1bc5b598674f24bbaf0714130bcf5b in term 0.
I20240502 02:10:10.174618 9471 leader_election.cc:302] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1bc5b598674f24bbaf0714130bcf5b, 8736c9ad53e8442fad4b8a577791aff8; no voters:
I20240502 02:10:10.174911 10016 raft_consensus.cc:2781] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:10:10.175006 10016 raft_consensus.cc:482] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:10:10.175092 10016 raft_consensus.cc:3037] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.176276 10016 raft_consensus.cc:504] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.176618 10016 leader_election.cc:288] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [CANDIDATE]: Term 1 election: Requested vote from peers f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194:36383), 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195:35865)
I20240502 02:10:10.177019 9673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c86c5791e074d3cae477c8d67077778" candidate_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6d70134a4212a44a0da8100aeb35"
I20240502 02:10:10.177083 9807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c86c5791e074d3cae477c8d67077778" candidate_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8736c9ad53e8442fad4b8a577791aff8"
I20240502 02:10:10.177157 9673 raft_consensus.cc:3037] T 3c86c5791e074d3cae477c8d67077778 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.177196 9807 raft_consensus.cc:3037] T 3c86c5791e074d3cae477c8d67077778 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.178480 9673 raft_consensus.cc:2454] T 3c86c5791e074d3cae477c8d67077778 P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a1bc5b598674f24bbaf0714130bcf5b in term 1.
I20240502 02:10:10.178828 9471 leader_election.cc:302] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1bc5b598674f24bbaf0714130bcf5b, f74e6d70134a4212a44a0da8100aeb35; no voters:
I20240502 02:10:10.178853 9807 raft_consensus.cc:2454] T 3c86c5791e074d3cae477c8d67077778 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a1bc5b598674f24bbaf0714130bcf5b in term 1.
I20240502 02:10:10.179036 10016 raft_consensus.cc:2781] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:10:10.179289 10016 raft_consensus.cc:686] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 LEADER]: Becoming Leader. State: Replica: 6a1bc5b598674f24bbaf0714130bcf5b, State: Running, Role: LEADER
I20240502 02:10:10.179626 10016 consensus_queue.cc:237] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.182982 9395 catalog_manager.cc:5484] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193). New cstate: current_term: 1 leader_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } health_report { overall_health: UNKNOWN } } }
W20240502 02:10:10.195693 9587 tablet.cc:2369] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 02:10:10.268289 10015 raft_consensus.cc:482] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:10:10.268427 10015 raft_consensus.cc:504] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.269471 10015 leader_election.cc:288] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619), 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195:35865)
I20240502 02:10:10.274361 9807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c63071e4946459fb633c2b8e8a6be78" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8736c9ad53e8442fad4b8a577791aff8" is_pre_election: true
I20240502 02:10:10.274523 9539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c63071e4946459fb633c2b8e8a6be78" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" is_pre_election: true
I20240502 02:10:10.274577 9807 raft_consensus.cc:2454] T 8c63071e4946459fb633c2b8e8a6be78 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 0.
I20240502 02:10:10.274755 9539 raft_consensus.cc:2454] T 8c63071e4946459fb633c2b8e8a6be78 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 0.
I20240502 02:10:10.274992 9605 leader_election.cc:302] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8736c9ad53e8442fad4b8a577791aff8, f74e6d70134a4212a44a0da8100aeb35; no voters:
I20240502 02:10:10.275279 10015 raft_consensus.cc:2781] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:10:10.275344 10015 raft_consensus.cc:482] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:10:10.275391 10015 raft_consensus.cc:3037] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.276208 10015 raft_consensus.cc:504] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.276583 10015 leader_election.cc:288] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 election: Requested vote from peers 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619), 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195:35865)
I20240502 02:10:10.276923 9539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c63071e4946459fb633c2b8e8a6be78" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b"
I20240502 02:10:10.276942 9807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c63071e4946459fb633c2b8e8a6be78" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8736c9ad53e8442fad4b8a577791aff8"
I20240502 02:10:10.277043 9807 raft_consensus.cc:3037] T 8c63071e4946459fb633c2b8e8a6be78 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.277040 9539 raft_consensus.cc:3037] T 8c63071e4946459fb633c2b8e8a6be78 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.277776 9807 raft_consensus.cc:2454] T 8c63071e4946459fb633c2b8e8a6be78 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 1.
I20240502 02:10:10.277935 9539 raft_consensus.cc:2454] T 8c63071e4946459fb633c2b8e8a6be78 P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 1.
I20240502 02:10:10.278077 9605 leader_election.cc:302] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8736c9ad53e8442fad4b8a577791aff8, f74e6d70134a4212a44a0da8100aeb35; no voters:
I20240502 02:10:10.278265 10015 raft_consensus.cc:2781] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:10:10.278357 10015 raft_consensus.cc:686] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [term 1 LEADER]: Becoming Leader. State: Replica: f74e6d70134a4212a44a0da8100aeb35, State: Running, Role: LEADER
I20240502 02:10:10.278712 10015 consensus_queue.cc:237] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.281916 9395 catalog_manager.cc:5484] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "f74e6d70134a4212a44a0da8100aeb35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } health_report { overall_health: UNKNOWN } } }
W20240502 02:10:10.318812 9855 tablet.cc:2369] T 3c86c5791e074d3cae477c8d67077778 P 8736c9ad53e8442fad4b8a577791aff8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 02:10:10.375804 10014 raft_consensus.cc:482] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:10:10.375977 10014 raft_consensus.cc:504] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } }
I20240502 02:10:10.377130 10014 leader_election.cc:288] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194:36383), 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619)
I20240502 02:10:10.380812 9673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb12390841324803a9b0e256a82735ad" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6d70134a4212a44a0da8100aeb35" is_pre_election: true
I20240502 02:10:10.380887 9539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb12390841324803a9b0e256a82735ad" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" is_pre_election: true
I20240502 02:10:10.380970 9673 raft_consensus.cc:2454] T fb12390841324803a9b0e256a82735ad P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 0.
I20240502 02:10:10.381018 9539 raft_consensus.cc:2454] T fb12390841324803a9b0e256a82735ad P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 0.
I20240502 02:10:10.381323 9738 leader_election.cc:302] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8736c9ad53e8442fad4b8a577791aff8, f74e6d70134a4212a44a0da8100aeb35; no voters:
I20240502 02:10:10.381598 10014 raft_consensus.cc:2781] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:10:10.381667 10014 raft_consensus.cc:482] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:10:10.381695 10014 raft_consensus.cc:3037] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.382612 10014 raft_consensus.cc:504] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } }
I20240502 02:10:10.382966 10014 leader_election.cc:288] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194:36383), 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619)
I20240502 02:10:10.383334 9673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb12390841324803a9b0e256a82735ad" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6d70134a4212a44a0da8100aeb35"
I20240502 02:10:10.383350 9539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb12390841324803a9b0e256a82735ad" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b"
I20240502 02:10:10.383447 9539 raft_consensus.cc:3037] T fb12390841324803a9b0e256a82735ad P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.383450 9673 raft_consensus.cc:3037] T fb12390841324803a9b0e256a82735ad P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.384204 9673 raft_consensus.cc:2454] T fb12390841324803a9b0e256a82735ad P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 1.
I20240502 02:10:10.384217 9539 raft_consensus.cc:2454] T fb12390841324803a9b0e256a82735ad P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 1.
I20240502 02:10:10.384518 9740 leader_election.cc:302] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1bc5b598674f24bbaf0714130bcf5b, 8736c9ad53e8442fad4b8a577791aff8; no voters:
I20240502 02:10:10.384685 10014 raft_consensus.cc:2781] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:10:10.384949 10014 raft_consensus.cc:686] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [term 1 LEADER]: Becoming Leader. State: Replica: 8736c9ad53e8442fad4b8a577791aff8, State: Running, Role: LEADER
I20240502 02:10:10.385274 10014 consensus_queue.cc:237] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } }
I20240502 02:10:10.388368 9395 catalog_manager.cc:5484] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "8736c9ad53e8442fad4b8a577791aff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } health_report { overall_health: UNKNOWN } } }
W20240502 02:10:10.550892 10035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:10:10.551199 10035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:10:10.554632 10035 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 02:10:10.599354 10016 consensus_queue.cc:1035] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:10.605166 10026 consensus_queue.cc:1035] T 3c86c5791e074d3cae477c8d67077778 P 6a1bc5b598674f24bbaf0714130bcf5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:10.652639 9807 raft_consensus.cc:1261] T 8c63071e4946459fb633c2b8e8a6be78 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Refusing update from remote peer f74e6d70134a4212a44a0da8100aeb35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:10:10.653051 9539 raft_consensus.cc:1261] T 8c63071e4946459fb633c2b8e8a6be78 P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Refusing update from remote peer f74e6d70134a4212a44a0da8100aeb35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:10:10.653080 10015 consensus_queue.cc:1035] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:10.653671 10015 consensus_queue.cc:1035] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:10.659426 9673 raft_consensus.cc:1261] T fb12390841324803a9b0e256a82735ad P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Refusing update from remote peer 8736c9ad53e8442fad4b8a577791aff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:10:10.659475 9539 raft_consensus.cc:1261] T fb12390841324803a9b0e256a82735ad P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Refusing update from remote peer 8736c9ad53e8442fad4b8a577791aff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:10:10.659933 10014 consensus_queue.cc:1035] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:10.660095 10033 consensus_queue.cc:1035] T fb12390841324803a9b0e256a82735ad P 8736c9ad53e8442fad4b8a577791aff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:10.664561 10060 mvcc.cc:205] Tried to move back new op lower bound from 7023066360428437504 to 7023066358904438784. Current Snapshot: MvccSnapshot[applied={T|T < 7023066360428437504}]
I20240502 02:10:10.668990 10075 mvcc.cc:205] Tried to move back new op lower bound from 7023066360428437504 to 7023066358904438784. Current Snapshot: MvccSnapshot[applied={T|T < 7023066360428437504}]
I20240502 02:10:10.673600 10059 mvcc.cc:205] Tried to move back new op lower bound from 7023066360428437504 to 7023066358904438784. Current Snapshot: MvccSnapshot[applied={T|T < 7023066360428437504}]
INSERT report
rows total: 2000
time total: 120.595 ms
time per row: 0.0602975 ms
I20240502 02:10:10.718755 9394 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43074:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 02:10:10.719332 9394 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 02:10:10.724690 9517 tablet_service.cc:1467] Processing CreateTablet for tablet 98de9c29108a49f8a2ff1b090c6712ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=41394a8650154522b5b3ce8c511858cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.724781 9653 tablet_service.cc:1467] Processing CreateTablet for tablet 98de9c29108a49f8a2ff1b090c6712ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=41394a8650154522b5b3ce8c511858cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.724936 9519 tablet_service.cc:1467] Processing CreateTablet for tablet 1f391f76ac144036a34f7bdb9e8db0d3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=41394a8650154522b5b3ce8c511858cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.725070 9517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98de9c29108a49f8a2ff1b090c6712ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.725140 9653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98de9c29108a49f8a2ff1b090c6712ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.725258 9519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f391f76ac144036a34f7bdb9e8db0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.725592 9518 tablet_service.cc:1467] Processing CreateTablet for tablet 907cafe653834ed0bb104bd0cf947164 (DEFAULT_TABLE table=rebalance_test_table_1 [id=41394a8650154522b5b3ce8c511858cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.725792 9518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907cafe653834ed0bb104bd0cf947164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.727741 9784 tablet_service.cc:1467] Processing CreateTablet for tablet 98de9c29108a49f8a2ff1b090c6712ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=41394a8650154522b5b3ce8c511858cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.728020 9784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98de9c29108a49f8a2ff1b090c6712ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.728541 9785 tablet_service.cc:1467] Processing CreateTablet for tablet 1f391f76ac144036a34f7bdb9e8db0d3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=41394a8650154522b5b3ce8c511858cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.728574 9652 tablet_service.cc:1467] Processing CreateTablet for tablet 907cafe653834ed0bb104bd0cf947164 (DEFAULT_TABLE table=rebalance_test_table_1 [id=41394a8650154522b5b3ce8c511858cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.728746 9785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f391f76ac144036a34f7bdb9e8db0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.728818 9652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907cafe653834ed0bb104bd0cf947164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.729020 10092 tablet_bootstrap.cc:492] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35: Bootstrap starting.
I20240502 02:10:10.729995 10092 tablet_bootstrap.cc:654] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.730393 9783 tablet_service.cc:1467] Processing CreateTablet for tablet 907cafe653834ed0bb104bd0cf947164 (DEFAULT_TABLE table=rebalance_test_table_1 [id=41394a8650154522b5b3ce8c511858cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.730787 9783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907cafe653834ed0bb104bd0cf947164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.731083 10093 tablet_bootstrap.cc:492] T 98de9c29108a49f8a2ff1b090c6712ac P 8736c9ad53e8442fad4b8a577791aff8: Bootstrap starting.
I20240502 02:10:10.731201 9653 tablet_service.cc:1467] Processing CreateTablet for tablet 1f391f76ac144036a34f7bdb9e8db0d3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=41394a8650154522b5b3ce8c511858cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:10.731431 9653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f391f76ac144036a34f7bdb9e8db0d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:10.731870 10093 tablet_bootstrap.cc:654] T 98de9c29108a49f8a2ff1b090c6712ac P 8736c9ad53e8442fad4b8a577791aff8: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.731873 10091 tablet_bootstrap.cc:492] T 98de9c29108a49f8a2ff1b090c6712ac P 6a1bc5b598674f24bbaf0714130bcf5b: Bootstrap starting.
I20240502 02:10:10.732888 10091 tablet_bootstrap.cc:654] T 98de9c29108a49f8a2ff1b090c6712ac P 6a1bc5b598674f24bbaf0714130bcf5b: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.733122 10092 tablet_bootstrap.cc:492] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35: No bootstrap required, opened a new log
I20240502 02:10:10.733122 10093 tablet_bootstrap.cc:492] T 98de9c29108a49f8a2ff1b090c6712ac P 8736c9ad53e8442fad4b8a577791aff8: No bootstrap required, opened a new log
I20240502 02:10:10.733209 10093 ts_tablet_manager.cc:1367] T 98de9c29108a49f8a2ff1b090c6712ac P 8736c9ad53e8442fad4b8a577791aff8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 02:10:10.733209 10092 ts_tablet_manager.cc:1367] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 02:10:10.733686 10092 raft_consensus.cc:348] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.733686 10093 raft_consensus.cc:348] T 98de9c29108a49f8a2ff1b090c6712ac P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.733794 10092 raft_consensus.cc:374] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.733794 10093 raft_consensus.cc:374] T 98de9c29108a49f8a2ff1b090c6712ac P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.733841 10093 raft_consensus.cc:729] T 98de9c29108a49f8a2ff1b090c6712ac P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8736c9ad53e8442fad4b8a577791aff8, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.733841 10092 raft_consensus.cc:729] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6d70134a4212a44a0da8100aeb35, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.733979 10092 consensus_queue.cc:260] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.733979 10093 consensus_queue.cc:260] T 98de9c29108a49f8a2ff1b090c6712ac P 8736c9ad53e8442fad4b8a577791aff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.734457 10093 ts_tablet_manager.cc:1398] T 98de9c29108a49f8a2ff1b090c6712ac P 8736c9ad53e8442fad4b8a577791aff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:10:10.734645 10093 tablet_bootstrap.cc:492] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8: Bootstrap starting.
I20240502 02:10:10.734669 10091 tablet_bootstrap.cc:492] T 98de9c29108a49f8a2ff1b090c6712ac P 6a1bc5b598674f24bbaf0714130bcf5b: No bootstrap required, opened a new log
I20240502 02:10:10.734737 10091 ts_tablet_manager.cc:1367] T 98de9c29108a49f8a2ff1b090c6712ac P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 02:10:10.735071 10091 raft_consensus.cc:348] T 98de9c29108a49f8a2ff1b090c6712ac P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.735146 10091 raft_consensus.cc:374] T 98de9c29108a49f8a2ff1b090c6712ac P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.735164 10091 raft_consensus.cc:729] T 98de9c29108a49f8a2ff1b090c6712ac P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1bc5b598674f24bbaf0714130bcf5b, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.735288 10091 consensus_queue.cc:260] T 98de9c29108a49f8a2ff1b090c6712ac P 6a1bc5b598674f24bbaf0714130bcf5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.735468 10093 tablet_bootstrap.cc:654] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.735556 10091 ts_tablet_manager.cc:1398] T 98de9c29108a49f8a2ff1b090c6712ac P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:10:10.735702 10091 tablet_bootstrap.cc:492] T 1f391f76ac144036a34f7bdb9e8db0d3 P 6a1bc5b598674f24bbaf0714130bcf5b: Bootstrap starting.
I20240502 02:10:10.736578 10091 tablet_bootstrap.cc:654] T 1f391f76ac144036a34f7bdb9e8db0d3 P 6a1bc5b598674f24bbaf0714130bcf5b: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.737850 10092 ts_tablet_manager.cc:1398] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 02:10:10.738071 10092 tablet_bootstrap.cc:492] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35: Bootstrap starting.
I20240502 02:10:10.738392 10091 tablet_bootstrap.cc:492] T 1f391f76ac144036a34f7bdb9e8db0d3 P 6a1bc5b598674f24bbaf0714130bcf5b: No bootstrap required, opened a new log
I20240502 02:10:10.738497 10091 ts_tablet_manager.cc:1367] T 1f391f76ac144036a34f7bdb9e8db0d3 P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 02:10:10.739152 10091 raft_consensus.cc:348] T 1f391f76ac144036a34f7bdb9e8db0d3 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.739267 10091 raft_consensus.cc:374] T 1f391f76ac144036a34f7bdb9e8db0d3 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.739300 10091 raft_consensus.cc:729] T 1f391f76ac144036a34f7bdb9e8db0d3 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1bc5b598674f24bbaf0714130bcf5b, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.739444 10092 tablet_bootstrap.cc:654] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.739465 10091 consensus_queue.cc:260] T 1f391f76ac144036a34f7bdb9e8db0d3 P 6a1bc5b598674f24bbaf0714130bcf5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.739764 10091 ts_tablet_manager.cc:1398] T 1f391f76ac144036a34f7bdb9e8db0d3 P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:10:10.739925 10091 tablet_bootstrap.cc:492] T 907cafe653834ed0bb104bd0cf947164 P 6a1bc5b598674f24bbaf0714130bcf5b: Bootstrap starting.
I20240502 02:10:10.740705 10092 tablet_bootstrap.cc:492] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35: No bootstrap required, opened a new log
I20240502 02:10:10.740731 10093 tablet_bootstrap.cc:492] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8: No bootstrap required, opened a new log
I20240502 02:10:10.740829 10093 ts_tablet_manager.cc:1367] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 02:10:10.740921 10092 ts_tablet_manager.cc:1367] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 02:10:10.741289 10091 tablet_bootstrap.cc:654] T 907cafe653834ed0bb104bd0cf947164 P 6a1bc5b598674f24bbaf0714130bcf5b: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.741325 10093 raft_consensus.cc:348] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.741436 10093 raft_consensus.cc:374] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.741478 10093 raft_consensus.cc:729] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8736c9ad53e8442fad4b8a577791aff8, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.741665 10093 consensus_queue.cc:260] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.741660 10092 raft_consensus.cc:348] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.741990 10093 ts_tablet_manager.cc:1398] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:10:10.742194 10093 tablet_bootstrap.cc:492] T 907cafe653834ed0bb104bd0cf947164 P 8736c9ad53e8442fad4b8a577791aff8: Bootstrap starting.
I20240502 02:10:10.742264 10092 raft_consensus.cc:374] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.742312 10092 raft_consensus.cc:729] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6d70134a4212a44a0da8100aeb35, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.742528 10092 consensus_queue.cc:260] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.742888 10092 ts_tablet_manager.cc:1398] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 02:10:10.743063 10092 tablet_bootstrap.cc:492] T 1f391f76ac144036a34f7bdb9e8db0d3 P f74e6d70134a4212a44a0da8100aeb35: Bootstrap starting.
I20240502 02:10:10.743125 10093 tablet_bootstrap.cc:654] T 907cafe653834ed0bb104bd0cf947164 P 8736c9ad53e8442fad4b8a577791aff8: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.744303 10091 tablet_bootstrap.cc:492] T 907cafe653834ed0bb104bd0cf947164 P 6a1bc5b598674f24bbaf0714130bcf5b: No bootstrap required, opened a new log
I20240502 02:10:10.744398 10091 ts_tablet_manager.cc:1367] T 907cafe653834ed0bb104bd0cf947164 P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 02:10:10.744427 10092 tablet_bootstrap.cc:654] T 1f391f76ac144036a34f7bdb9e8db0d3 P f74e6d70134a4212a44a0da8100aeb35: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:10.744951 10093 tablet_bootstrap.cc:492] T 907cafe653834ed0bb104bd0cf947164 P 8736c9ad53e8442fad4b8a577791aff8: No bootstrap required, opened a new log
I20240502 02:10:10.744887 10091 raft_consensus.cc:348] T 907cafe653834ed0bb104bd0cf947164 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.745009 10091 raft_consensus.cc:374] T 907cafe653834ed0bb104bd0cf947164 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.745043 10093 ts_tablet_manager.cc:1367] T 907cafe653834ed0bb104bd0cf947164 P 8736c9ad53e8442fad4b8a577791aff8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 02:10:10.745050 10091 raft_consensus.cc:729] T 907cafe653834ed0bb104bd0cf947164 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1bc5b598674f24bbaf0714130bcf5b, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.745194 10091 consensus_queue.cc:260] T 907cafe653834ed0bb104bd0cf947164 P 6a1bc5b598674f24bbaf0714130bcf5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.745456 10091 ts_tablet_manager.cc:1398] T 907cafe653834ed0bb104bd0cf947164 P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:10:10.745462 10093 raft_consensus.cc:348] T 907cafe653834ed0bb104bd0cf947164 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.745549 10093 raft_consensus.cc:374] T 907cafe653834ed0bb104bd0cf947164 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.745586 10093 raft_consensus.cc:729] T 907cafe653834ed0bb104bd0cf947164 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8736c9ad53e8442fad4b8a577791aff8, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.745705 10093 consensus_queue.cc:260] T 907cafe653834ed0bb104bd0cf947164 P 8736c9ad53e8442fad4b8a577791aff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.745965 10093 ts_tablet_manager.cc:1398] T 907cafe653834ed0bb104bd0cf947164 P 8736c9ad53e8442fad4b8a577791aff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:10:10.747826 10092 tablet_bootstrap.cc:492] T 1f391f76ac144036a34f7bdb9e8db0d3 P f74e6d70134a4212a44a0da8100aeb35: No bootstrap required, opened a new log
I20240502 02:10:10.747896 10092 ts_tablet_manager.cc:1367] T 1f391f76ac144036a34f7bdb9e8db0d3 P f74e6d70134a4212a44a0da8100aeb35: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 02:10:10.748255 10092 raft_consensus.cc:348] T 1f391f76ac144036a34f7bdb9e8db0d3 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.748346 10092 raft_consensus.cc:374] T 1f391f76ac144036a34f7bdb9e8db0d3 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:10.748379 10092 raft_consensus.cc:729] T 1f391f76ac144036a34f7bdb9e8db0d3 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6d70134a4212a44a0da8100aeb35, State: Initialized, Role: FOLLOWER
I20240502 02:10:10.748504 10092 consensus_queue.cc:260] T 1f391f76ac144036a34f7bdb9e8db0d3 P f74e6d70134a4212a44a0da8100aeb35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.748750 10092 ts_tablet_manager.cc:1398] T 1f391f76ac144036a34f7bdb9e8db0d3 P f74e6d70134a4212a44a0da8100aeb35: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:10:10.762490 10019 raft_consensus.cc:482] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:10:10.762632 10019 raft_consensus.cc:504] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.763046 10019 leader_election.cc:288] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619), 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195:35865)
I20240502 02:10:10.763537 9539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98de9c29108a49f8a2ff1b090c6712ac" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" is_pre_election: true
I20240502 02:10:10.763566 9806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98de9c29108a49f8a2ff1b090c6712ac" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8736c9ad53e8442fad4b8a577791aff8" is_pre_election: true
I20240502 02:10:10.763718 9539 raft_consensus.cc:2454] T 98de9c29108a49f8a2ff1b090c6712ac P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 0.
I20240502 02:10:10.763717 9806 raft_consensus.cc:2454] T 98de9c29108a49f8a2ff1b090c6712ac P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 0.
I20240502 02:10:10.764045 9607 leader_election.cc:302] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1bc5b598674f24bbaf0714130bcf5b, f74e6d70134a4212a44a0da8100aeb35; no voters:
I20240502 02:10:10.764282 10019 raft_consensus.cc:2781] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:10:10.764355 10019 raft_consensus.cc:482] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:10:10.764408 10019 raft_consensus.cc:3037] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.765298 10019 raft_consensus.cc:504] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.765655 10019 leader_election.cc:288] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 election: Requested vote from peers 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619), 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195:35865)
I20240502 02:10:10.766042 9539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98de9c29108a49f8a2ff1b090c6712ac" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b"
I20240502 02:10:10.766042 9806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98de9c29108a49f8a2ff1b090c6712ac" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8736c9ad53e8442fad4b8a577791aff8"
I20240502 02:10:10.766177 9806 raft_consensus.cc:3037] T 98de9c29108a49f8a2ff1b090c6712ac P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.766177 9539 raft_consensus.cc:3037] T 98de9c29108a49f8a2ff1b090c6712ac P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.767235 9806 raft_consensus.cc:2454] T 98de9c29108a49f8a2ff1b090c6712ac P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 1.
I20240502 02:10:10.767235 9539 raft_consensus.cc:2454] T 98de9c29108a49f8a2ff1b090c6712ac P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 1.
I20240502 02:10:10.767557 9607 leader_election.cc:302] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1bc5b598674f24bbaf0714130bcf5b, f74e6d70134a4212a44a0da8100aeb35; no voters:
I20240502 02:10:10.767747 10019 raft_consensus.cc:2781] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:10:10.767838 10019 raft_consensus.cc:686] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [term 1 LEADER]: Becoming Leader. State: Replica: f74e6d70134a4212a44a0da8100aeb35, State: Running, Role: LEADER
I20240502 02:10:10.767997 10019 consensus_queue.cc:237] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.769697 9394 catalog_manager.cc:5484] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "f74e6d70134a4212a44a0da8100aeb35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } health_report { overall_health: UNKNOWN } } }
I20240502 02:10:10.902479 10019 raft_consensus.cc:482] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:10:10.902645 10019 raft_consensus.cc:504] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.903084 10019 leader_election.cc:288] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619), 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195:35865)
I20240502 02:10:10.903632 9806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907cafe653834ed0bb104bd0cf947164" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8736c9ad53e8442fad4b8a577791aff8" is_pre_election: true
I20240502 02:10:10.903632 9539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907cafe653834ed0bb104bd0cf947164" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" is_pre_election: true
I20240502 02:10:10.903793 9539 raft_consensus.cc:2454] T 907cafe653834ed0bb104bd0cf947164 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 0.
I20240502 02:10:10.903793 9806 raft_consensus.cc:2454] T 907cafe653834ed0bb104bd0cf947164 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 0.
I20240502 02:10:10.904138 9607 leader_election.cc:302] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1bc5b598674f24bbaf0714130bcf5b, f74e6d70134a4212a44a0da8100aeb35; no voters:
I20240502 02:10:10.904328 10019 raft_consensus.cc:2781] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:10:10.904393 10019 raft_consensus.cc:482] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:10:10.904420 10019 raft_consensus.cc:3037] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.905256 10019 raft_consensus.cc:504] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.905598 10019 leader_election.cc:288] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 election: Requested vote from peers 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619), 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195:35865)
I20240502 02:10:10.905972 9806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907cafe653834ed0bb104bd0cf947164" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8736c9ad53e8442fad4b8a577791aff8"
I20240502 02:10:10.905969 9539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907cafe653834ed0bb104bd0cf947164" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b"
I20240502 02:10:10.906083 9806 raft_consensus.cc:3037] T 907cafe653834ed0bb104bd0cf947164 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.906096 9539 raft_consensus.cc:3037] T 907cafe653834ed0bb104bd0cf947164 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.906951 9806 raft_consensus.cc:2454] T 907cafe653834ed0bb104bd0cf947164 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 1.
I20240502 02:10:10.906983 9539 raft_consensus.cc:2454] T 907cafe653834ed0bb104bd0cf947164 P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 1.
I20240502 02:10:10.907261 9605 leader_election.cc:302] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8736c9ad53e8442fad4b8a577791aff8, f74e6d70134a4212a44a0da8100aeb35; no voters:
I20240502 02:10:10.907409 10019 raft_consensus.cc:2781] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:10:10.907496 10019 raft_consensus.cc:686] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [term 1 LEADER]: Becoming Leader. State: Replica: f74e6d70134a4212a44a0da8100aeb35, State: Running, Role: LEADER
I20240502 02:10:10.907647 10019 consensus_queue.cc:237] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.908392 10033 raft_consensus.cc:482] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:10:10.908499 10033 raft_consensus.cc:504] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.908893 10033 leader_election.cc:288] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619), f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194:36383)
I20240502 02:10:10.909230 9539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f391f76ac144036a34f7bdb9e8db0d3" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" is_pre_election: true
I20240502 02:10:10.909229 9673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f391f76ac144036a34f7bdb9e8db0d3" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6d70134a4212a44a0da8100aeb35" is_pre_election: true
I20240502 02:10:10.909343 9539 raft_consensus.cc:2454] T 1f391f76ac144036a34f7bdb9e8db0d3 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 0.
I20240502 02:10:10.909341 9673 raft_consensus.cc:2454] T 1f391f76ac144036a34f7bdb9e8db0d3 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 0.
I20240502 02:10:10.909618 9738 leader_election.cc:302] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8736c9ad53e8442fad4b8a577791aff8, f74e6d70134a4212a44a0da8100aeb35; no voters:
I20240502 02:10:10.909775 10033 raft_consensus.cc:2781] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:10:10.909821 10033 raft_consensus.cc:482] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:10:10.909855 10033 raft_consensus.cc:3037] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.910588 10033 raft_consensus.cc:504] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.910872 10033 leader_election.cc:288] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 election: Requested vote from peers 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619), f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194:36383)
I20240502 02:10:10.911197 9673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f391f76ac144036a34f7bdb9e8db0d3" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6d70134a4212a44a0da8100aeb35"
I20240502 02:10:10.911329 9673 raft_consensus.cc:3037] T 1f391f76ac144036a34f7bdb9e8db0d3 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.911219 9394 catalog_manager.cc:5484] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "f74e6d70134a4212a44a0da8100aeb35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } health_report { overall_health: UNKNOWN } } }
I20240502 02:10:10.912220 9673 raft_consensus.cc:2454] T 1f391f76ac144036a34f7bdb9e8db0d3 P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 1.
I20240502 02:10:10.912447 9539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f391f76ac144036a34f7bdb9e8db0d3" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b"
I20240502 02:10:10.912513 9738 leader_election.cc:302] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8736c9ad53e8442fad4b8a577791aff8, f74e6d70134a4212a44a0da8100aeb35; no voters:
I20240502 02:10:10.912568 9539 raft_consensus.cc:3037] T 1f391f76ac144036a34f7bdb9e8db0d3 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:10.912676 10033 raft_consensus.cc:2781] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:10:10.912767 10033 raft_consensus.cc:686] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 LEADER]: Becoming Leader. State: Replica: 8736c9ad53e8442fad4b8a577791aff8, State: Running, Role: LEADER
I20240502 02:10:10.912886 10033 consensus_queue.cc:237] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:10.913547 9539 raft_consensus.cc:2454] T 1f391f76ac144036a34f7bdb9e8db0d3 P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 1.
I20240502 02:10:10.914598 9394 catalog_manager.cc:5484] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "8736c9ad53e8442fad4b8a577791aff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } health_report { overall_health: HEALTHY } } }
W20240502 02:10:11.035696 10106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:10:11.035957 10106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:10:11.039094 10106 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 02:10:11.137437 9807 raft_consensus.cc:1261] T 907cafe653834ed0bb104bd0cf947164 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Refusing update from remote peer f74e6d70134a4212a44a0da8100aeb35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:10:11.137657 9539 raft_consensus.cc:1261] T 98de9c29108a49f8a2ff1b090c6712ac P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Refusing update from remote peer f74e6d70134a4212a44a0da8100aeb35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:10:11.137863 9538 raft_consensus.cc:1261] T 907cafe653834ed0bb104bd0cf947164 P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Refusing update from remote peer f74e6d70134a4212a44a0da8100aeb35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:10:11.137437 9806 raft_consensus.cc:1261] T 98de9c29108a49f8a2ff1b090c6712ac P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Refusing update from remote peer f74e6d70134a4212a44a0da8100aeb35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:10:11.138978 10015 consensus_queue.cc:1035] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:11.139298 10015 consensus_queue.cc:1035] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:11.141180 9538 raft_consensus.cc:1261] T 1f391f76ac144036a34f7bdb9e8db0d3 P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Refusing update from remote peer 8736c9ad53e8442fad4b8a577791aff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:10:11.141809 9673 raft_consensus.cc:1261] T 1f391f76ac144036a34f7bdb9e8db0d3 P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Refusing update from remote peer 8736c9ad53e8442fad4b8a577791aff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:10:11.141842 10014 consensus_queue.cc:1035] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:11.142482 10019 consensus_queue.cc:1035] T 98de9c29108a49f8a2ff1b090c6712ac P f74e6d70134a4212a44a0da8100aeb35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:11.142776 10019 consensus_queue.cc:1035] T 907cafe653834ed0bb104bd0cf947164 P f74e6d70134a4212a44a0da8100aeb35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:11.144183 10014 consensus_queue.cc:1035] T 1f391f76ac144036a34f7bdb9e8db0d3 P 8736c9ad53e8442fad4b8a577791aff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 116.065 ms
time per row: 0.0580325 ms
I20240502 02:10:11.204384 9388 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43074:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 02:10:11.204962 9388 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 02:10:11.210453 9653 tablet_service.cc:1467] Processing CreateTablet for tablet 38eaff5e32444d33982608e4b682ed51 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19c890b1ad3743acbaac4b1d426adfa4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:11.210471 9783 tablet_service.cc:1467] Processing CreateTablet for tablet 38eaff5e32444d33982608e4b682ed51 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19c890b1ad3743acbaac4b1d426adfa4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:11.210443 9518 tablet_service.cc:1467] Processing CreateTablet for tablet 38eaff5e32444d33982608e4b682ed51 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19c890b1ad3743acbaac4b1d426adfa4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:11.210788 9653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38eaff5e32444d33982608e4b682ed51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:11.210788 9783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38eaff5e32444d33982608e4b682ed51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:11.210795 9518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38eaff5e32444d33982608e4b682ed51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:11.210978 9785 tablet_service.cc:1467] Processing CreateTablet for tablet d818a2e86a2949b2affee1ded8699167 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19c890b1ad3743acbaac4b1d426adfa4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:11.211234 9785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d818a2e86a2949b2affee1ded8699167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:11.213124 10092 tablet_bootstrap.cc:492] T 38eaff5e32444d33982608e4b682ed51 P f74e6d70134a4212a44a0da8100aeb35: Bootstrap starting.
I20240502 02:10:11.213950 9653 tablet_service.cc:1467] Processing CreateTablet for tablet d818a2e86a2949b2affee1ded8699167 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19c890b1ad3743acbaac4b1d426adfa4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:11.214118 10092 tablet_bootstrap.cc:654] T 38eaff5e32444d33982608e4b682ed51 P f74e6d70134a4212a44a0da8100aeb35: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:11.214210 9653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d818a2e86a2949b2affee1ded8699167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:11.214326 9652 tablet_service.cc:1467] Processing CreateTablet for tablet a01fa18f1b534632b8dd73290225d9f6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19c890b1ad3743acbaac4b1d426adfa4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:11.214593 9652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01fa18f1b534632b8dd73290225d9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:11.214550 9785 tablet_service.cc:1467] Processing CreateTablet for tablet a01fa18f1b534632b8dd73290225d9f6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19c890b1ad3743acbaac4b1d426adfa4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:11.214726 10093 tablet_bootstrap.cc:492] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8: Bootstrap starting.
I20240502 02:10:11.214800 9785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01fa18f1b534632b8dd73290225d9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:11.214900 10091 tablet_bootstrap.cc:492] T 38eaff5e32444d33982608e4b682ed51 P 6a1bc5b598674f24bbaf0714130bcf5b: Bootstrap starting.
I20240502 02:10:11.215826 10091 tablet_bootstrap.cc:654] T 38eaff5e32444d33982608e4b682ed51 P 6a1bc5b598674f24bbaf0714130bcf5b: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:11.217280 10091 tablet_bootstrap.cc:492] T 38eaff5e32444d33982608e4b682ed51 P 6a1bc5b598674f24bbaf0714130bcf5b: No bootstrap required, opened a new log
I20240502 02:10:11.217293 10093 tablet_bootstrap.cc:654] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:11.217382 10091 ts_tablet_manager.cc:1367] T 38eaff5e32444d33982608e4b682ed51 P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 02:10:11.217425 9518 tablet_service.cc:1467] Processing CreateTablet for tablet d818a2e86a2949b2affee1ded8699167 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19c890b1ad3743acbaac4b1d426adfa4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:11.217705 9518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d818a2e86a2949b2affee1ded8699167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:11.217854 10091 raft_consensus.cc:348] T 38eaff5e32444d33982608e4b682ed51 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.217970 10091 raft_consensus.cc:374] T 38eaff5e32444d33982608e4b682ed51 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:11.218004 10091 raft_consensus.cc:729] T 38eaff5e32444d33982608e4b682ed51 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1bc5b598674f24bbaf0714130bcf5b, State: Initialized, Role: FOLLOWER
I20240502 02:10:11.218147 10091 consensus_queue.cc:260] T 38eaff5e32444d33982608e4b682ed51 P 6a1bc5b598674f24bbaf0714130bcf5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.218429 10093 tablet_bootstrap.cc:492] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8: No bootstrap required, opened a new log
I20240502 02:10:11.218474 10091 ts_tablet_manager.cc:1398] T 38eaff5e32444d33982608e4b682ed51 P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:10:11.218508 10093 ts_tablet_manager.cc:1367] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 02:10:11.218631 10092 tablet_bootstrap.cc:492] T 38eaff5e32444d33982608e4b682ed51 P f74e6d70134a4212a44a0da8100aeb35: No bootstrap required, opened a new log
I20240502 02:10:11.218711 10092 ts_tablet_manager.cc:1367] T 38eaff5e32444d33982608e4b682ed51 P f74e6d70134a4212a44a0da8100aeb35: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240502 02:10:11.218926 10093 raft_consensus.cc:348] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.219050 10093 raft_consensus.cc:374] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:11.219084 10093 raft_consensus.cc:729] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8736c9ad53e8442fad4b8a577791aff8, State: Initialized, Role: FOLLOWER
I20240502 02:10:11.219071 10092 raft_consensus.cc:348] T 38eaff5e32444d33982608e4b682ed51 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.219179 10092 raft_consensus.cc:374] T 38eaff5e32444d33982608e4b682ed51 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:11.219215 10092 raft_consensus.cc:729] T 38eaff5e32444d33982608e4b682ed51 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6d70134a4212a44a0da8100aeb35, State: Initialized, Role: FOLLOWER
I20240502 02:10:11.219187 10093 consensus_queue.cc:260] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.219338 10092 consensus_queue.cc:260] T 38eaff5e32444d33982608e4b682ed51 P f74e6d70134a4212a44a0da8100aeb35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.219460 10093 ts_tablet_manager.cc:1398] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:10:11.219621 10092 ts_tablet_manager.cc:1398] T 38eaff5e32444d33982608e4b682ed51 P f74e6d70134a4212a44a0da8100aeb35: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 02:10:11.219640 10093 tablet_bootstrap.cc:492] T d818a2e86a2949b2affee1ded8699167 P 8736c9ad53e8442fad4b8a577791aff8: Bootstrap starting.
I20240502 02:10:11.219785 10092 tablet_bootstrap.cc:492] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35: Bootstrap starting.
I20240502 02:10:11.220604 10092 tablet_bootstrap.cc:654] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:11.220849 10093 tablet_bootstrap.cc:654] T d818a2e86a2949b2affee1ded8699167 P 8736c9ad53e8442fad4b8a577791aff8: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:11.221345 9517 tablet_service.cc:1467] Processing CreateTablet for tablet a01fa18f1b534632b8dd73290225d9f6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19c890b1ad3743acbaac4b1d426adfa4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:10:11.221623 9517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01fa18f1b534632b8dd73290225d9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:10:11.222740 10092 tablet_bootstrap.cc:492] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35: No bootstrap required, opened a new log
I20240502 02:10:11.222846 10092 ts_tablet_manager.cc:1367] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 02:10:11.223627 10092 raft_consensus.cc:348] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.223830 10092 raft_consensus.cc:374] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:11.223860 10091 tablet_bootstrap.cc:492] T d818a2e86a2949b2affee1ded8699167 P 6a1bc5b598674f24bbaf0714130bcf5b: Bootstrap starting.
I20240502 02:10:11.223937 10092 raft_consensus.cc:729] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6d70134a4212a44a0da8100aeb35, State: Initialized, Role: FOLLOWER
I20240502 02:10:11.224123 10092 consensus_queue.cc:260] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.224483 10092 ts_tablet_manager.cc:1398] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 02:10:11.224853 10092 tablet_bootstrap.cc:492] T a01fa18f1b534632b8dd73290225d9f6 P f74e6d70134a4212a44a0da8100aeb35: Bootstrap starting.
I20240502 02:10:11.224884 10091 tablet_bootstrap.cc:654] T d818a2e86a2949b2affee1ded8699167 P 6a1bc5b598674f24bbaf0714130bcf5b: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:11.226013 10131 raft_consensus.cc:482] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:10:11.226245 10092 tablet_bootstrap.cc:654] T a01fa18f1b534632b8dd73290225d9f6 P f74e6d70134a4212a44a0da8100aeb35: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:11.226269 10093 tablet_bootstrap.cc:492] T d818a2e86a2949b2affee1ded8699167 P 8736c9ad53e8442fad4b8a577791aff8: No bootstrap required, opened a new log
I20240502 02:10:11.226176 10131 raft_consensus.cc:504] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.226356 10093 ts_tablet_manager.cc:1367] T d818a2e86a2949b2affee1ded8699167 P 8736c9ad53e8442fad4b8a577791aff8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 02:10:11.226799 10131 leader_election.cc:288] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619), 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195:35865)
I20240502 02:10:11.226935 10091 tablet_bootstrap.cc:492] T d818a2e86a2949b2affee1ded8699167 P 6a1bc5b598674f24bbaf0714130bcf5b: No bootstrap required, opened a new log
I20240502 02:10:11.226891 10093 raft_consensus.cc:348] T d818a2e86a2949b2affee1ded8699167 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.227008 10093 raft_consensus.cc:374] T d818a2e86a2949b2affee1ded8699167 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:11.227020 10091 ts_tablet_manager.cc:1367] T d818a2e86a2949b2affee1ded8699167 P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 02:10:11.227044 10093 raft_consensus.cc:729] T d818a2e86a2949b2affee1ded8699167 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8736c9ad53e8442fad4b8a577791aff8, State: Initialized, Role: FOLLOWER
I20240502 02:10:11.227180 10093 consensus_queue.cc:260] T d818a2e86a2949b2affee1ded8699167 P 8736c9ad53e8442fad4b8a577791aff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.227300 9807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d818a2e86a2949b2affee1ded8699167" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8736c9ad53e8442fad4b8a577791aff8" is_pre_election: true
I20240502 02:10:11.227388 9807 raft_consensus.cc:2454] T d818a2e86a2949b2affee1ded8699167 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 0.
I20240502 02:10:11.227456 10093 ts_tablet_manager.cc:1398] T d818a2e86a2949b2affee1ded8699167 P 8736c9ad53e8442fad4b8a577791aff8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 02:10:11.227478 10091 raft_consensus.cc:348] T d818a2e86a2949b2affee1ded8699167 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.227593 10091 raft_consensus.cc:374] T d818a2e86a2949b2affee1ded8699167 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:11.227630 10091 raft_consensus.cc:729] T d818a2e86a2949b2affee1ded8699167 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1bc5b598674f24bbaf0714130bcf5b, State: Initialized, Role: FOLLOWER
I20240502 02:10:11.227634 10093 tablet_bootstrap.cc:492] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8: Bootstrap starting.
I20240502 02:10:11.227666 9605 leader_election.cc:302] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8736c9ad53e8442fad4b8a577791aff8, f74e6d70134a4212a44a0da8100aeb35; no voters:
I20240502 02:10:11.227746 10091 consensus_queue.cc:260] T d818a2e86a2949b2affee1ded8699167 P 6a1bc5b598674f24bbaf0714130bcf5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.228080 10091 ts_tablet_manager.cc:1398] T d818a2e86a2949b2affee1ded8699167 P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:10:11.228248 10091 tablet_bootstrap.cc:492] T a01fa18f1b534632b8dd73290225d9f6 P 6a1bc5b598674f24bbaf0714130bcf5b: Bootstrap starting.
I20240502 02:10:11.228932 10093 tablet_bootstrap.cc:654] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:11.228935 10131 raft_consensus.cc:2781] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:10:11.228991 10131 raft_consensus.cc:482] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:10:11.229036 10131 raft_consensus.cc:3037] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:11.229432 10091 tablet_bootstrap.cc:654] T a01fa18f1b534632b8dd73290225d9f6 P 6a1bc5b598674f24bbaf0714130bcf5b: Neither blocks nor log segments found. Creating new log.
I20240502 02:10:11.228031 9538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d818a2e86a2949b2affee1ded8699167" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" is_pre_election: true
I20240502 02:10:11.230230 10131 raft_consensus.cc:504] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.230262 9538 raft_consensus.cc:2454] T d818a2e86a2949b2affee1ded8699167 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 0.
I20240502 02:10:11.230675 10131 leader_election.cc:288] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 election: Requested vote from peers 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619), 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195:35865)
I20240502 02:10:11.231030 9807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d818a2e86a2949b2affee1ded8699167" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8736c9ad53e8442fad4b8a577791aff8"
I20240502 02:10:11.231160 9807 raft_consensus.cc:3037] T d818a2e86a2949b2affee1ded8699167 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:11.231492 9538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d818a2e86a2949b2affee1ded8699167" candidate_uuid: "f74e6d70134a4212a44a0da8100aeb35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b"
I20240502 02:10:11.231607 9538 raft_consensus.cc:3037] T d818a2e86a2949b2affee1ded8699167 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:11.232251 9807 raft_consensus.cc:2454] T d818a2e86a2949b2affee1ded8699167 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 1.
I20240502 02:10:11.232451 10092 tablet_bootstrap.cc:492] T a01fa18f1b534632b8dd73290225d9f6 P f74e6d70134a4212a44a0da8100aeb35: No bootstrap required, opened a new log
I20240502 02:10:11.232539 10092 ts_tablet_manager.cc:1367] T a01fa18f1b534632b8dd73290225d9f6 P f74e6d70134a4212a44a0da8100aeb35: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240502 02:10:11.232542 9605 leader_election.cc:302] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8736c9ad53e8442fad4b8a577791aff8, f74e6d70134a4212a44a0da8100aeb35; no voters:
I20240502 02:10:11.232652 9538 raft_consensus.cc:2454] T d818a2e86a2949b2affee1ded8699167 P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6d70134a4212a44a0da8100aeb35 in term 1.
I20240502 02:10:11.232908 10131 raft_consensus.cc:2781] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:10:11.233042 10092 raft_consensus.cc:348] T a01fa18f1b534632b8dd73290225d9f6 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.233098 10131 raft_consensus.cc:686] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [term 1 LEADER]: Becoming Leader. State: Replica: f74e6d70134a4212a44a0da8100aeb35, State: Running, Role: LEADER
I20240502 02:10:11.233142 10092 raft_consensus.cc:374] T a01fa18f1b534632b8dd73290225d9f6 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:11.233175 10092 raft_consensus.cc:729] T a01fa18f1b534632b8dd73290225d9f6 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6d70134a4212a44a0da8100aeb35, State: Initialized, Role: FOLLOWER
I20240502 02:10:11.233233 10131 consensus_queue.cc:237] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.233270 10092 consensus_queue.cc:260] T a01fa18f1b534632b8dd73290225d9f6 P f74e6d70134a4212a44a0da8100aeb35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.233538 10091 tablet_bootstrap.cc:492] T a01fa18f1b534632b8dd73290225d9f6 P 6a1bc5b598674f24bbaf0714130bcf5b: No bootstrap required, opened a new log
I20240502 02:10:11.233553 10092 ts_tablet_manager.cc:1398] T a01fa18f1b534632b8dd73290225d9f6 P f74e6d70134a4212a44a0da8100aeb35: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 02:10:11.233639 10091 ts_tablet_manager.cc:1367] T a01fa18f1b534632b8dd73290225d9f6 P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 02:10:11.234216 10093 tablet_bootstrap.cc:492] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8: No bootstrap required, opened a new log
I20240502 02:10:11.234305 10093 ts_tablet_manager.cc:1367] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 02:10:11.234328 10091 raft_consensus.cc:348] T a01fa18f1b534632b8dd73290225d9f6 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.234457 10091 raft_consensus.cc:374] T a01fa18f1b534632b8dd73290225d9f6 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:11.234496 10091 raft_consensus.cc:729] T a01fa18f1b534632b8dd73290225d9f6 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1bc5b598674f24bbaf0714130bcf5b, State: Initialized, Role: FOLLOWER
I20240502 02:10:11.234650 10091 consensus_queue.cc:260] T a01fa18f1b534632b8dd73290225d9f6 P 6a1bc5b598674f24bbaf0714130bcf5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.234798 10093 raft_consensus.cc:348] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.234901 10093 raft_consensus.cc:374] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:10:11.234932 10093 raft_consensus.cc:729] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8736c9ad53e8442fad4b8a577791aff8, State: Initialized, Role: FOLLOWER
I20240502 02:10:11.235010 10091 ts_tablet_manager.cc:1398] T a01fa18f1b534632b8dd73290225d9f6 P 6a1bc5b598674f24bbaf0714130bcf5b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:10:11.235045 10093 consensus_queue.cc:260] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.235363 10093 ts_tablet_manager.cc:1398] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 02:10:11.235313 9394 catalog_manager.cc:5484] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194). New cstate: current_term: 1 leader_uuid: "f74e6d70134a4212a44a0da8100aeb35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } health_report { overall_health: UNKNOWN } } }
I20240502 02:10:11.335664 10132 raft_consensus.cc:482] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:10:11.335822 10132 raft_consensus.cc:504] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.336278 10132 leader_election.cc:288] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619), f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194:36383)
I20240502 02:10:11.336807 9538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eaff5e32444d33982608e4b682ed51" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" is_pre_election: true
I20240502 02:10:11.336848 9673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eaff5e32444d33982608e4b682ed51" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6d70134a4212a44a0da8100aeb35" is_pre_election: true
I20240502 02:10:11.336969 9538 raft_consensus.cc:2454] T 38eaff5e32444d33982608e4b682ed51 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 0.
I20240502 02:10:11.336987 9673 raft_consensus.cc:2454] T 38eaff5e32444d33982608e4b682ed51 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 0.
I20240502 02:10:11.337386 9740 leader_election.cc:302] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1bc5b598674f24bbaf0714130bcf5b, 8736c9ad53e8442fad4b8a577791aff8; no voters:
I20240502 02:10:11.337575 10132 raft_consensus.cc:2781] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:10:11.337635 10132 raft_consensus.cc:482] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:10:11.337662 10132 raft_consensus.cc:3037] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:11.338694 10132 raft_consensus.cc:504] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.339113 10132 leader_election.cc:288] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 election: Requested vote from peers 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619), f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194:36383)
I20240502 02:10:11.339542 9673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eaff5e32444d33982608e4b682ed51" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6d70134a4212a44a0da8100aeb35"
I20240502 02:10:11.339669 9673 raft_consensus.cc:3037] T 38eaff5e32444d33982608e4b682ed51 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:11.340464 9673 raft_consensus.cc:2454] T 38eaff5e32444d33982608e4b682ed51 P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 1.
I20240502 02:10:11.340802 9738 leader_election.cc:302] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8736c9ad53e8442fad4b8a577791aff8, f74e6d70134a4212a44a0da8100aeb35; no voters:
I20240502 02:10:11.340786 9538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eaff5e32444d33982608e4b682ed51" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b"
I20240502 02:10:11.340921 9538 raft_consensus.cc:3037] T 38eaff5e32444d33982608e4b682ed51 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:11.340998 10132 raft_consensus.cc:2781] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:10:11.341145 10132 raft_consensus.cc:686] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 LEADER]: Becoming Leader. State: Replica: 8736c9ad53e8442fad4b8a577791aff8, State: Running, Role: LEADER
I20240502 02:10:11.341300 10132 consensus_queue.cc:237] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.341989 9538 raft_consensus.cc:2454] T 38eaff5e32444d33982608e4b682ed51 P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 1.
I20240502 02:10:11.343315 9388 catalog_manager.cc:5484] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "8736c9ad53e8442fad4b8a577791aff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } health_report { overall_health: HEALTHY } } }
I20240502 02:10:11.582320 10132 raft_consensus.cc:482] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:10:11.582494 10132 raft_consensus.cc:504] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.582980 10132 leader_election.cc:288] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619), f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194:36383)
I20240502 02:10:11.583468 9538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01fa18f1b534632b8dd73290225d9f6" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" is_pre_election: true
I20240502 02:10:11.583609 9538 raft_consensus.cc:2454] T a01fa18f1b534632b8dd73290225d9f6 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 0.
I20240502 02:10:11.583593 9673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01fa18f1b534632b8dd73290225d9f6" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6d70134a4212a44a0da8100aeb35" is_pre_election: true
I20240502 02:10:11.583779 9673 raft_consensus.cc:2454] T a01fa18f1b534632b8dd73290225d9f6 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 0.
I20240502 02:10:11.583962 9740 leader_election.cc:302] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1bc5b598674f24bbaf0714130bcf5b, 8736c9ad53e8442fad4b8a577791aff8; no voters:
I20240502 02:10:11.584162 10132 raft_consensus.cc:2781] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:10:11.584215 10132 raft_consensus.cc:482] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:10:11.584241 10132 raft_consensus.cc:3037] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:11.585074 10132 raft_consensus.cc:504] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.585435 10132 leader_election.cc:288] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 election: Requested vote from peers 6a1bc5b598674f24bbaf0714130bcf5b (127.27.111.193:37619), f74e6d70134a4212a44a0da8100aeb35 (127.27.111.194:36383)
I20240502 02:10:11.585795 9538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01fa18f1b534632b8dd73290225d9f6" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1bc5b598674f24bbaf0714130bcf5b"
I20240502 02:10:11.585825 9673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01fa18f1b534632b8dd73290225d9f6" candidate_uuid: "8736c9ad53e8442fad4b8a577791aff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6d70134a4212a44a0da8100aeb35"
I20240502 02:10:11.585922 9538 raft_consensus.cc:3037] T a01fa18f1b534632b8dd73290225d9f6 P 6a1bc5b598674f24bbaf0714130bcf5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:11.585954 9673 raft_consensus.cc:3037] T a01fa18f1b534632b8dd73290225d9f6 P f74e6d70134a4212a44a0da8100aeb35 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:10:11.586863 9673 raft_consensus.cc:2454] T a01fa18f1b534632b8dd73290225d9f6 P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 1.
I20240502 02:10:11.586861 9538 raft_consensus.cc:2454] T a01fa18f1b534632b8dd73290225d9f6 P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8736c9ad53e8442fad4b8a577791aff8 in term 1.
I20240502 02:10:11.587162 9738 leader_election.cc:302] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8736c9ad53e8442fad4b8a577791aff8, f74e6d70134a4212a44a0da8100aeb35; no voters:
I20240502 02:10:11.587324 10132 raft_consensus.cc:2781] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:10:11.587414 10132 raft_consensus.cc:686] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [term 1 LEADER]: Becoming Leader. State: Replica: 8736c9ad53e8442fad4b8a577791aff8, State: Running, Role: LEADER
I20240502 02:10:11.587563 10132 consensus_queue.cc:237] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } }
I20240502 02:10:11.589259 9388 catalog_manager.cc:5484] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8736c9ad53e8442fad4b8a577791aff8 (127.27.111.195). New cstate: current_term: 1 leader_uuid: "8736c9ad53e8442fad4b8a577791aff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } health_report { overall_health: HEALTHY } } }
I20240502 02:10:11.632539 10131 consensus_queue.cc:1035] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:11.635972 10019 consensus_queue.cc:1035] T d818a2e86a2949b2affee1ded8699167 P f74e6d70134a4212a44a0da8100aeb35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:11.749920 10132 consensus_queue.cc:1035] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:11.753626 10132 consensus_queue.cc:1035] T 38eaff5e32444d33982608e4b682ed51 P 8736c9ad53e8442fad4b8a577791aff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 02:10:11.794298 10156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:10:11.794632 10156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:10:11.797979 10156 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 02:10:11.894380 9538 raft_consensus.cc:1261] T a01fa18f1b534632b8dd73290225d9f6 P 6a1bc5b598674f24bbaf0714130bcf5b [term 1 FOLLOWER]: Refusing update from remote peer 8736c9ad53e8442fad4b8a577791aff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:10:11.895054 9673 raft_consensus.cc:1261] T a01fa18f1b534632b8dd73290225d9f6 P f74e6d70134a4212a44a0da8100aeb35 [term 1 FOLLOWER]: Refusing update from remote peer 8736c9ad53e8442fad4b8a577791aff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:10:11.896342 10014 consensus_queue.cc:1035] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:10:11.899250 10132 consensus_queue.cc:1035] T a01fa18f1b534632b8dd73290225d9f6 P 8736c9ad53e8442fad4b8a577791aff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 117.722 ms
time per row: 0.0588612 ms
I20240502 02:10:11.960788 28095 external_mini_cluster.cc:1326] Running /tmp/dist-test-task43FDSt/build/debug/bin/kudu
/tmp/dist-test-task43FDSt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.111.196:44107
--local_ip_for_outbound_sockets=127.27.111.196
--tserver_master_addrs=127.27.111.254:39465
--webserver_port=42551
--webserver_interface=127.27.111.196
--builtin_ntp_servers=127.27.111.212:46539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 02:10:12.071841 10204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:10:12.072080 10204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:10:12.072126 10204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:10:12.075384 10204 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:10:12.075472 10204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:10:12.075548 10204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.111.196
I20240502 02:10:12.079305 10204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.111.212:46539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.111.196:44107
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.27.111.196
--webserver_port=42551
--tserver_master_addrs=127.27.111.254:39465
--never_fsync=true
--heap_profile_path=/tmp/kudu.10204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.111.196
--log_dir=/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 01:43:14 UTC on 1a15035a6830
build id 81
I20240502 02:10:12.080267 10204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:10:12.081440 10204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:10:12.089700 10209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:10:12.090310 10210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:10:12.090517 10212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:10:12.091096 10204 server_base.cc:1034] running on GCE node
I20240502 02:10:12.091490 10204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:10:12.092051 10204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:10:12.093204 10204 hybrid_clock.cc:648] HybridClock initialized: now 1714615812093185 us; error 26 us; skew 500 ppm
I20240502 02:10:12.095216 10204 webserver.cc:460] Webserver started at http://127.27.111.196:42551/ using document root <none> and password file <none>
I20240502 02:10:12.095793 10204 fs_manager.cc:359] Metadata directory not provided
I20240502 02:10:12.095865 10204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:10:12.099274 10204 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20240502 02:10:12.101084 10218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:12.102041 10204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 02:10:12.102144 10204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5cb7c03f646743a0b5662b2a7529395b"
format_stamp: "Formatted at 2024-05-02 02:10:06 on dist-test-slave-0pw1"
I20240502 02:10:12.102602 10204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:10:12.117691 10204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 02:10:12.118330 10204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:10:12.118523 10204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:10:12.119107 10204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:10:12.120002 10204 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:10:12.120049 10204 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:12.120093 10204 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:10:12.120127 10204 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 02:10:12.129099 10204 rpc_server.cc:307] RPC server started. Bound to: 127.27.111.196:44107
I20240502 02:10:12.129130 10333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.111.196:44107 every 8 connection(s)
I20240502 02:10:12.130215 10204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task43FDSt/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714615075754784-28095-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 02:10:12.137301 28095 external_mini_cluster.cc:1388] Started /tmp/dist-test-task43FDSt/build/debug/bin/kudu as pid 10204
I20240502 02:10:12.139581 10334 heartbeater.cc:347] Connected to a master server at 127.27.111.254:39465
I20240502 02:10:12.139880 10334 heartbeater.cc:464] Registering TS with master...
I20240502 02:10:12.140514 10334 heartbeater.cc:511] Master 127.27.111.254:39465 requested a full tablet report, sending...
I20240502 02:10:12.141842 9388 ts_manager.cc:194] Re-registered known tserver with Master: 5cb7c03f646743a0b5662b2a7529395b (127.27.111.196:44107)
I20240502 02:10:12.142472 9388 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.111.196:52207
I20240502 02:10:12.367231 9652 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:12.368163 9518 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:10:12.369658 10267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:10:12.370627 9785 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:12.468381 10267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:10:12.468662 9518 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:10:12.470434 9785 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:12.478374 9652 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:12.580968 10267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:10:12.581180 9518 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:10:12.581642 9785 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:12.586624 9652 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:12.667971 10267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:10:12.671110 9785 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:12.673712 9518 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:10:12.678920 9652 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:10:12.716827 9672 consensus_queue.cc:237] T 8c63071e4946459fb633c2b8e8a6be78 P f74e6d70134a4212a44a0da8100aeb35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6a1bc5b598674f24bbaf0714130bcf5b" member_type: VOTER last_known_addr { host: "127.27.111.193" port: 37619 } } peers { permanent_uuid: "f74e6d70134a4212a44a0da8100aeb35" member_type: VOTER last_known_addr { host: "127.27.111.194" port: 36383 } } peers { permanent_uuid: "8736c9ad53e8442fad4b8a577791aff8" member_type: VOTER last_known_addr { host: "127.27.111.195" port: 35865 } attrs { replace: true } } peers { permanent_uuid: "5cb7c03f646743a0b5662b2a7529395b" member_type: NON_VOTER last_known_addr { host: "127.27.111.196"